JR East ATOS System Display/Emulator (departure and announcements + LED signage + Bonus coding lessons)

Don't want to see this ad? Sign up for anRPF Premium Membershiptoday. Support the community. Stop the ads.


rworne

Member
The ESP-8266, in my case the ESP-01, is a postage-stamp size bit of electronics that contains everything you need to add wireless networking to a project. They are cheap too - when I bought mine, they were around $5 each. They are even less now.

ESP8266-01 1-600x315.jpg


Able to work as a wireless client or as a wi-fi access point, I have had a ton of fun with these in the past, the best one was making a captive portal for nosy neighbors who wonder what the "FBI SURVEILLANCE VAN" was that showed up on their list of wireless networks. When they connected to it, it showed a convincing login page with scary legalese and played back a loop of recorded random police scanner traffic.

Usually I use them for more mundane activities - another variant called the WiFi Kit 8 is about the size of a thumb drive and as thick as a stick of gum. It has a built-in OLED display that displays a list of active WiFi networks whenever you apply 5V power to it.

So back to the ESP8266.

For as nice as these things are, they are not friendly to hook up. To me, friendly is the ability to use it with a breadboard - adapters exist for this, but they were not available when I ordered them, and I don't have any, so there you go.

The ESP-01 likes 3.3V, so it is not 5V friendly. So when pairing it up to an Arduino, Teensy, or what have you, you'll need to either convert the voltages from 5V to 3.3V, or just do what I do and get a 3.3V Arduino or Teensy.

So how easy is this thing to use? Pretty easy:

C:
#include <ESP8266WiFi.h>

char ssid[] = "MY NETWORK";  //  your network SSID (name)
char pass[] = "MY NETWORK PASSWORD";       // your network password

void setup()
{
  WiFi.begin(ssid, pass);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
  }

}

That's it. All the code necessary to start up and connect to a WiFi network.

Setting up an access point isn't that much different:

C:
#include <ESP8266WiFi.h>

char ssid[] = "MY NETWORK";  //  your network SSID (name)
char pass[] = "MY NETWORK PASSWORD";       // your network password

void setup()
{
  WiFi.softAP(ssid, pass);
}

All that in a few lines of code, amazing.

But that's just the setup, now you need to do something with it.... I needed to sync with a time source - Internet, GPS, I don't care how, but I have these ESP8266s sitting around waiting to be put to good use. So NTP it is. Rather than re-invent the wheel, I looked for existing solutions, and there was:

ARDUINO TIME SYNC FROM NTP SERVER USING ESP8266 WIFI MODULE

The code from here for the ESP8266 was used unmodified aside from making a blinking LED to tell me the ESP8266 is actually running.

The Teensy's Serial1 was used by the WAV Trigger, so Serial2 is used instead.

On the Matrix board, I modified what was supplied at the site above into a function:

C:
void syncTime()
{
  char buffer[40];
  int buffer_loop = 0;
  TimeChangeRule *tcr;        // pointer to the time change rule, use to get the TZ abbrev
  utc = now();

  time_t old_t;

  while (Serial2.available()) {
    buffer[buffer_loop++] = Serial2.read();
    dataSync = true;
  }
  if (dataSync == true) {
    if ((buffer[0] == 'U') && (buffer[1] == 'N') && (buffer[2] == 'X')) {
      // if data sent is the UNX token, take it
      unixString[0] = buffer[3];
      unixString[1] = buffer[4];
      unixString[2] = buffer[5];
      unixString[3] = buffer[6];
      unixString[4] = buffer[7];
      unixString[5] = buffer[8];
      unixString[6] = buffer[9];
      unixString[7] = buffer[10];
      unixString[8] = buffer[11];
      unixString[9] = buffer[12];
      unixString[10] = '\0';

      // print the UNX time on the UNO serial
      Serial.println();
      Serial.print("TIME FROM ESP: ");
      Serial.print(unixString[0]);
      Serial.print(unixString[1]);
      Serial.print(unixString[2]);
      Serial.print(unixString[3]);
      Serial.print(unixString[4]);
      Serial.print(unixString[5]);
      Serial.print(unixString[6]);
      Serial.print(unixString[7]);
      Serial.print(unixString[8]);
      Serial.print(unixString[9]);
      Serial.println();

      unixTime = atol(unixString);

      // Synchronize the time with the internal clock
      // for external use RTC.setTime();

      old_t = now();

      if ((abs(unixTime - old_t) < 7200) || !timeSync)  //sync if time is recent (within 2 hours) or we just had power applied
      {
        setTime(unixTime);                              //set current time to UNIX time from NTP server
        Teensy3Clock.set(now());                         //set RTC to current time
        utc = now();                                     //now pull the current time
        pacific = usPacific.toLocal(utc, &tcr);         //and load it into the time change rule.
        if (usPacific.utcIsDST(utc))                     //check for DST and print out results
          printDateTime(usPDT, utc, " Los Angeles");
        else
          printDateTime(usPST, utc, " Los Angeles");
        timeSync = true;
      }
      else
      {
        Serial.print("Sync error: delta = ");             // if difference is too great, print the difference
        Serial.println(abs(unixTime - old_t));
      }
      dataSync = false;
    }
  }
}

This is called from the main loop(). If it finds a message waiting - that means the time sync signal (which comes in every few minutes, I need to slow it down a bit) has arrived. The ESP8266 runs independently from the Teensy I am using, so it can take up to a minute or so for the time to sync properly.

My modifications are:
1. Do not sync the time if a ludicrous result arrives. Ignore any time change more than 2 hours (7200 seconds) unless we are starting up.
2. Set a flag when the time syncs for the first time - we always sync in that case.
3. Sync the Teensy's RTC with the received time
4. Get the local time (PDT and PST) appropriately.

Gotta love that Timezone Library, it made life a lot easier, and I know the precise time in my area:

Code:
18:36:44 Fri 08 May 2020 PDT Los Angeles

And there it is:


Blinking blue light means it's still alive and working.
 
Last edited:

Don't want to see this ad? Sign up for anRPF Premium Membershiptoday. Support the community. Stop the ads.

rworne

Member
The other part of the is project is the audio, which is currently handled by the WAV Trigger:
WAVTriggerBeauty.jpg

This is a very useful board that allows for some really cool tricks. I originally purchased this for a Space Battleship Yamato build, that 1/350 scale monster set, when I ran into a dead end debating my skills trying to airbrush paint and weather a model set (for the first time) on a $300+ kit.

But everything else was done - music, updated audio effects, LED guns, motor driven cannons... you can control it with the original IR gun controller or hook it up to an app on your phone and control it over bluetooth.

Like other projects I do, I doubt my model making capabilities or run into build issues I cannot solve at the time and abandon them.

Another example:
Know those 6-8" high arcade game miniatures that became available in the past 2 years or so? I've done this before. 500+ MAME games, plus Dragon's Lair, Space Ace, and all the other laserdisc based games. Doesn't sound like much now, but I had all the components for this working back in 2008 to fit in a 6-inch cabinet. Why did it die? I had difficulty fabricating a 6" cabinet, and the paint gave me problems, ruining my woodwork.

So on the back burner it went.

So with this lying around, what can it do for me? It's way overkill for my needs.
1. Play up to 4096 WAV files in CD-quality stereo audio.
2. Up to 14 channels of polyphony <- this was important for the Yamato build.
3. MIDI-capable
4. On-board amplifier
5. Software controlled, or can be hardwired to play back certain sounds via contact closure.

Only drawback is that it's big. Roughly 2.5"x2.5" in size.

I really wanted it to work with the DFPlayer - a rather inexpensive MP3 player that looks like this:
51JL8-FMAaL._AC_SY355_.jpg

Some problems when testing this one. MP3 files were slow to load, so WAV files were tried instead. These also loaded slowly, leaving unnatural gaps in the voice clippings. It's also difficult to determine reliably when the audio of one clip has finished playing to start the next one. But it's great for playing individual files. So no on this.

So the SD card is loaded up with 700-800 audio clips, wavTrigger library loaded into the Arduino IDE. Time to initialize it:

C:
#include <wavTrigger.h>

wavTrigger wTrig;                             // Our WAV Trigger object
char gWTrigVersion[VERSION_STRING_LEN];     // WAV Trigger version string
int  gNumTracks;                              // Number of tracks on SD card

void setup() {
  // WAV Trigger startup at 57600
  wTrig.start();

  delay(10);
  wTrig.setReporting(true);                    // this enables feedback of WAV Trigger status

  // Send a stop-all command and reset the sample-rate offset, in case we have
  //  reset while the WAV Trigger was already playing.
  wTrig.stopAllTracks();
  wTrig.samplerateOffset(0);

  // Allow time for the WAV Trigger to respond with the version string and
  //  number of tracks.
  delay(100);

  // If bi-directional communication is wired up, then we should by now be able
  //  to fetch the version string and number of tracks on the SD card.
  if (wTrig.getVersion(gWTrigVersion, VERSION_STRING_LEN)) {
    Serial.print(gWTrigVersion);
    Serial.print("\n");
    gNumTracks = wTrig.getNumTracks();
    Serial.print("Number of tracks = ");
    Serial.print(gNumTracks);
    Serial.print("\n");
  }
  else
  {
    Serial.print("WAV Trigger response not available");
  }
}

That gives you something like this on the debug output:
Code:
WAV Trigger v1.32
Number of tracks = 724

And we are all ready to go.

Next up is playing audio. You can just tell it to start playing, or queue up your tracks then tell them to start playing. I prefer the latter:

C:
enum audio_type
{
...
  A_ZEROJI = 301,
  A_JYUU = 302,
  A_NIJYUU = 303,
  A_ICHIJI = 304,
  A_NIJI = 305,
  A_SANJI = 306,
  A_YOJI = 307,
  A_GOJI = 308,
  A_ROKUJI = 309,
  A_SHICHIJI = 310,
  A_HACHIJI = 311,
  A_KUUJI = 312,
  A_JYUUJI = 313,
  A_NIJYUUJI = 314,
...
};
void speak_time(int hours, int mins)
{
int track;
  switch (hours)
  {
    case 0:
      track = A_REIJI;
      break;
    case 10:
      track = A_JYUUJI;
      break;
    case 20:
      track = A_NIJYUUJI;
      break;
    default:
      if (hours > 10 && hours < 20)
        track = A_JYUU;
      else if (hours > 20 && hours < 30)
        track = A_NIJYUU;
      else
        track =  0;
      break;
  }
  if (track)
  {
    wTrig.trackLoad(track);
    wTrig.trackResume(track);
    wTrig.update();
    delay(50);
    while (wTrig.isTrackPlaying(track))
    {
      wTrig.update();
      delay(50);
    }
  }
}

This is sample code from the time announcement function. We pass it the time in hours and minutes and it tears it apart into separate components compatible with the Japanese language, which has a regular and easily understood counting system - until you realize they have a unit of 10,000 and 100,000,000 and everything goes from readily understood to mental math exercises in a hurry.

Chiemi_Blouson-P1.jpg

Blouson Chiemi, a Japanese comedienne has a gag that gives advice for lovelorn females - "Does the flower chase the bee? No, it waits." - "When your chewing gum starts to lose its flavor, do you keep on chewing it? No, you eat a new piece of gum." - “Do you know how many men there in this world? 35 oku.” yeah, the old "more fish in the sea" gag is funny (or Japanese comedy is an acquired taste), but it took me a good 10 seconds or so to verify the number I came up with in my head was correct - 3.5 billion.

But telling time is easy in Japanese.

So in the sample above, we need to parse the hour from a 0-23 format. So for the first part it needs to say "0 'o'clock", "10 o'clock", or "20 o'clock" if the hour is exactly 12AM, 10AM or 8PM, otherwise, it needs to say just "10" or "20". In Japanese, they don't have numbers like "eleven" or "thirteen", these are counted similar to how we count numbers greater than 20 in English: "eleven" is "ten-one", "thirteen" is "ten-three", etc. Numbers greater than 19 follow a similar pattern: 21 is "two-ten-one" and 53 is "five-ten-three" - think of the two-ten as "twenty" and the five-ten as "fifty" and teens as erm... "tenty" and you pretty much got it. So now you have numbers down, now you need to understand counting units. "ji" is the counting unit for hours, it can also mean "hemorrhoids" - context is key here... so 10 o'clock is jyuu-ji, 3PM is 15:00 or jyuugo-ji. Minutes is "-fun" with the "u" pronounced as a short "oo" sound. But soft consonants like the "f" have their own special pronunciation rules in Japanese, can can change to a "b" or a double-"p" sound depending on the previous syllable. But enough of Japanese-101.

Each of the sound files are indexed.

So 10 o'clock "十時" is A_JYUUJI, which is equal to 313 in the enum above, so the WAV Trigger is asked to play back track #313, or 313_10ji.wav on the SD card. If nothing is found, it sets track to zero:

C:
  if (track)
  {
    wTrig.trackLoad(track);
    wTrig.trackResume(track);
    wTrig.update();
    delay(50);
    while (wTrig.isTrackPlaying(track))
    {
      wTrig.update();
      delay(50);
    }
  }

This bit of code handles the playback. If the track is zero, it just skips it and goes on to the next bit of code. If the track is a non-zero number, load the track and start playing (resume) it.

Then the WAV Trigger is told to send an update, and a delay to receive the update.
And just repeat those last two steps until the WAV trigger reports it is no longer playing a file (playback is finished).

At this point, you can process and load the next file for the rest of the hour announcement.

C:
  switch (hours % 10)
  {
    case 1:
      track = A_ICHIJI;
      break;
    case 2:
      track = A_NIJI;
      break;
    case 3:
      track = A_SANJI;
      break;
    case 4:
      track = A_YOJI;
      break;
    case 5:
      track = A_GOJI;
      break;
    case 6:
      track = A_ROKUJI;
      break;
    case 7:
      track = A_SHICHIJI;
      break;
    case 8:
      track = A_HACHIJI;
      break;
    case 9:
      track = A_KUUJI;
      break;
    default:
      track = 0;
  }
  if (track)
  {
    wTrig.trackLoad(track);
    wTrig.trackResume(track);
    wTrig.update();
    delay(50);
    while (wTrig.isTrackPlaying(track))
    {
      wTrig.update();
      delay(50);
    }
  }

and so on with the minutes handled in a similar fashion.

So all this is fine and dandy, but if I'm playing back a 15-second assembled audio file, how do I do that while keeping my lights and such going?
unnamed.jpg



to be continued...
 
Last edited:

rworne

Member
Threads are the equivalent of patting one's head and rubbing their belly at the same time.
data-star-trek-pat-head-rub-belly.gif


This is the one thing that will solve many a programmer's problems. it does that by creating a whole new bunch of even more esoteric problems whose sole purpose is to make your life miserable.

So how to bring this up without mentioning cores, scheduling, time slicing, context switching, deadlocks, semaphores, mutexes, and other terms crammed into my head in college?
We'll see.

A Teensy and Arduino are simple single-core CPU's. By default, they run as a single-threaded system - that is, one continuous process of executing instructions defined in your software. If you have two CPU's (or cores), you can run two simultaneous threads. A lot of modern PCs have up to 4 cores now, but your computer can evidently run a lot more than 4 things at once, so how does it do that?

Now I have to use the egghead terms above.

In simple terms, the computer just takes two (or more) tasks, and switches between them very quickly, picking up where it left off every time it switches. Same goes for the Teensy, and to some extent, an Arduino. The key thing to remember is while both tasks are running simultaneously, the device is actually just doing one thing at a time switching so fast it appears it is doing them at the same time.

When I last worked with Arduino, there wasn't a real thread library, but a library existed that worked similar to one. It could run multiple tasks at the same time, but you had to be careful to keep it from doing too much at once, as it was more or less *cooperative* multitasking and not *pre-emptive* multitasking. The difference? In cooperative multitasking threads need to give up unused resources and pass control off to other threads. This is sort of a crappy multitasking - think early MacOS or Windows 3.1. Pre-emptive multitasking switches threads whether the thread wants to or not, so other threads (and your main program) aren't starved of CPU time. This is why modern OS's (AmigaOS, Linux, OS/2, Windows NT, or OS X) run so much better than Windows 3.1, Windows 95-98 (16-bit programs), DesqView, and MacOS.

What I'll show you is threads on a Teensy, and later on an Arduino, if I remember where I put the code.

A very simple thread example:
C:
#include <TeensyThreads.h>

const int LED = 13;

void blinkthread() {
  while(1) {
    digitalWrite(LED, HIGH);
    threads.delay(150);
    digitalWrite(LED, LOW);
    threads.delay(150);
    threads.yield();
  }
}

void setup()
{
  pinMode(LED, OUTPUT);
  threads.addThread(blinkthread);
}

void loop() {

// do whatever else you want here

}

What does this do? Takes a loop that blinks an LED and places it into a separate thread. Once you kick off the thread, it will run forever - need to blink another LED? Do it in the main loop, do it in another thread, or add the code to the existing thread. I use these to kick off my audio events. I fire off a thread with the stuff I want to play, and it plays it in the background. When it finishes, it terminates itself. Handy-dandy.

But I mentioned earlier that threads are where dragons and other monsters lurk - true, and it's very easy to screw something up if you are not careful. Consider this:
C:
void setup()
{
  pinMode(LED, OUTPUT);
}

void loop() {

  threads.addThread(blinkthread);

}

This is a one-way express ticket to crashville. Every time the loop executes (which is a lot), it spawns another thread. In a computer there are two types of memory: a stack and a heap with a bunch of free memory in between. Think of them as matter and anti-matter, they should never touch each other. When the computer switches between threads, it needs to save the state information in the stack, which continuously grows as threads are added. Add a lot of threads and your stack grows considerably, not to mention the overhead that robs you of CPU time to do what you want it to do. Eventually they touch, and game over.

So lets add a conditional to it, so it only launches a thread when you want it to:
Code:
void setup()
{
  pinMode(LED, OUTPUT);
}

void loop() {
if(button_pushed)
{
  threads.addThread(blinkthread);
}
}

(I'm lazy and didn't code up the button code)
When you run this, it does nothing. Push the button once, the LED starts blinking: blink...blink...blink...
Push it again, and you'll notice it changed: blink.blink..blink.blink..blink.blink..
Push it a third time, and the pattern changes again.

You can get away with this for quite a few pushes as the pattern changes.

This seems cool, but what you have is two (or more) threads fighting over a resource (the LED). In this case, it may be intended. But in most of the cases where this happens, you DO NOT want it to happen. One good example is a thread writing to memory while another is reading the information being written by the other thread. Everything goes great for a while until both happen at the same time, and then you get a Seth Brundle moment with your data and it glitches out. To avoid this we need a mutex, that acts sort of as a turnstile for your threads - and which is just too much for me to type about at... 1AM.

Mostly those of you still with me would be interested in firing off sounds from a DFPlayer from a button push while your blinky light effects run along unhindered. What I showed you today in this post and the previous one is all the info you need to do this.

For Arduino, look up the ArduinoThread library. This will also get you what you want, and it works similarly:
C:
#include <Thread.h>
int ledPin = 13;

//My simple Thread
Thread myThread = Thread();
static bool ledStatus = false;

// callback for myThread
void blinkenlights(){
    ledStatus = !ledStatus; // alternates LED every time this function is called
    digitalWrite(ledPin, ledStatus);
}

void setup(){
    myThread.onRun(blinkenlights);  //set up a thread that calls "blinkenlights"
    myThread.setInterval(500);     // should run every 500ms
}

void loop(){
    // checks if thread should run
    if(myThread.shouldRun())      //need to manually check if thread should run
        myThread.run();           //if so, run the thread

    // Do other stuff...
    int x = 0;
    x = 1 + 2;
}

The key difference here is you need to manage the time. If you want a smooth blink once per second, you need to make sure that the other code you are running will finish in less than 500ms or your thread call will be late and you'll have irregular/delayed blinking. This becomes more of a problem the more threads you add, but if you keep everything efficient, you can juggle a few threads with no problems. Let's look at a wholly faked-up example:

We have Thread1 that needs to run 1x per second.
We have our main program.

Thread1 takes 200ms to complete.
The main thread takes 500ms to complete.
Awesome, we have 300ms to spare, so our thread should work, right? Not so fast...

C:
              Time     Elapsed Time
Thread1 runs: 200ms       0ms
Main runs:    500ms     200ms
Main runs:    500ms     700ms
Thread1 runs: 200ms    1200ms
Main runs:    500ms    1400ms
Main runs     500ms    1900ms
Thread1 runs: 200ms    2400ms

We can see the thread (example blink LED) though scheduled to blink every 1000ms, is actually blinking every 1200ms. Why? Because when the main thread checks to see if the thread is scheduled on the next loop, it sees it is 300ms early, so it skips it and runs the rest of the main thread which uses up another 500ms of time. On the next pass, Thread1 is late, so it runs, followed by the main thread. This pattern then repeats. So keep your threads short. In real-life you need to worry about 10's of milliseconds resolution in the main loop, so these issues do not crop up for slow events like navigation lights unless you are doing something rather time-consuming on one or more of your other threads. Key take away: long sequences should be put in threads, leaving the main loop to be as short as possible.

You can fix the example above to work with a simple timer. If main loop can run 1x per second you can get the time off the clock T1 when the main loop starts. Run Thread1 call and the rest of the main loop. Get the time again T2 at the end of the loop and calculate the elapsed time TE = T2 - T1. Now take the wait time TW = 1 sec - TE. This should be about 300ms. Then just put a delay in the end of the loop for time period TW. I don't mean calculate it by hand, put it in the code:

C:
#include <Thread.h>
int ledPin = 13;
unsigned long T1 = 0;

//My simple Thread
Thread myThread = Thread();
static bool ledStatus = false;

// callback for myThread
void blinkenlights(){
    ledStatus = !ledStatus; // alternates LED every time this function is called
    digitalWrite(ledPin, ledStatus);
}

void setup(){
    myThread.onRun(blinkenlights);  //set up a thread that calls "blinkenlights"
    myThread.setInterval(500);     // should run every 500ms
}

void loop(){
    T1 = millis();
    // checks if thread should run
    if(myThread.shouldRun())      //need to manually check if thread should run
        myThread.run();           //if so, run the thread

    // Do other stuff...
    int x = 0;
    x = 1 + 2;
    unsigned long T2 = millis();
    unsigned long TE = T2 - T1;
    unsigned long TW = 1000 - TE;  // 1000 is the time in milliseconds. This loop will run 1x/sec
    delay(TW);
}
You can shorten the last 4 lines to this:
C:
delay(1000-(millis() - T1));

Suddenly, everything is working again.
C:
               Time     Elapsed Time
Thread1 runs:  200ms       0ms
Main runs:     500ms     200ms
Delay Time:    300ms     700ms
Thread1 runs:  200ms    1000ms
Main runs:     500ms    1200ms
Delay Time:    300ms    1700ms
Thread1 runs   200ms    2000ms
Main runs:     500ms    2200ms

It works! As an added bonus, as you further modify your code, it continues to work!

One thing to watch out for is the execution time of your thread. You want your thread to finish before calling it again - or your audio playback would at best glitch out like Max Headroom or at worst, crash your program. So if you have a 1.5 second playback thread with the above 1 second main processing loop, you'd have at best the last 0.5 seconds of the audio cut off in a repeating loop. There are other ways to avoid it, but I'm worried enough this is too much info as it is.

In my project, trains come in at a minimum interval of 1 minute. My longest announcements are 20 seconds or so long, so I'm guaranteed that the thread will finish before it is called again and I do not need to enable any additional protection to prevent multiple calls to the same thread.

You can hear of code to libraries being "thread-safe". This means that it can be used in threads without worry about the frequency of how much it is called, and you do not have to worry about the read-while-writing problem I mentioned earlier. Pretty much every Arduino library is NOT thread-safe. So heed my warnings: Here be Dragons!

Keep your timings in check, prevent multiple calls before the thread exits, and you can keep things simple - have one thread for LEDS, have one for audio, etc.
 
Last edited:

rworne

Member
The IR Remote arrived yesterday, so now it's integrated and I can change options in the sign without recompiling and reloading:


Currently, it allows me to walk through the stations, changing the display with the up/down buttons to show a live presentation of the signage in use at that location on the top half of the screen.

Soon, left and right will scroll through the different lines.

The bottom half displays the station, direction, and track #.

Confirming the choice with the OK button turns off the menu and resumes the display from the selected location.
 

rworne

Member
IR Remotes:

These can always be of use, and on Amazon at least, can be had for under $5 - for the remote, an IR transmitter, and the IR receiver.

There are several libraries out there to use, but I'll stick with the IRRemote library, because it's compatible with the Teensy.
infrared-ir-wireless-remote-control-module-kit-arduino-raspberry-stelectronics-1609-20-STElect...jpg


This is pretty much what comes with it. The pinout of the receiver is (pins facing you, component side up, left to right):
GND, VCC, and OUT. Ground can go to ground, VCC to 3.3 or 5V, it doesn't care, and OUT to an unused pin.

Code to get it working is pretty simple:

C:
#include <boarddefs.h>
#include <IRremoteInt.h>
#include <IRremote.h>

const int RECV_PIN = 15; // Set this to the pin# you are using
IRrecv irrecv(RECV_PIN);
decode_results results;

enum nec_type
{
  NEC_1 = 0xFFA25D,
  NEC_2 = 0xFF629D,
  NEC_3 = 0xFFE21D,
  NEC_4 = 0xFF22DD,
  NEC_5 = 0xFF02FD,
  NEC_6 = 0xFFC23D,
  NEC_7 = 0xFFE01F,
  NEC_8 = 0xFFA857,
  NEC_9 = 0xFF906F,
  NEC_0 = 0xFF9867,
  NEC_S = 0xFF6897,
  NEC_P = 0xFFB04F,
  NEC_U = 0xFF18E7,
  NEC_L = 0xFF10EF,
  NEC_R = 0xFF5AA5,
  NEC_D = 0xFF4AB5,
  NEC_O = 0xFF38C7
};

const char * nec_decoder(int value)
{
  switch (value)
  {
    case NEC_1:
      return "1";
      break;
    case NEC_2:
      return "2";
      break;
    case NEC_3:
      return "3";
      break;
    case NEC_4:
      return "4";
      break;
    case NEC_5:
      return "5";
      break;
    case NEC_6:
      return "6";
      break;
    case NEC_7:
      return "7";
      break;
    case NEC_8:
      return "8";
      break;
    case NEC_9:
      return "9";
      break;
    case NEC_0:
      return "0";
      break;
    case NEC_S:
      return "*";
      break;
    case NEC_P:
      return "#";
      break;
    case NEC_U:
      return "U";
      break;
    case NEC_L:
      return "L";
      break;
    case NEC_R:
      return "R";
      break;
    case NEC_D:
      return "D";
      break;
    case NEC_O:
      return "O";
      break;
  }
  return "E";
}

void setup() {
  Serial.begin(9600);

  irrecv.enableIRIn();
  irrecv.blink13(true);
}

void loop() {
  if (irrecv.decode(&results)) {
    if (results.decode_type == NEC) {
      if (results.value != REPEAT) {
        Serial.print("Received NEC: ");
        Serial.print(results.value, HEX);
        Serial.print(" ");
        Serial.println(nec_decoder(results.value));
      }
    }
    else
    {
      Serial.print("Received Unknown: ");
      Serial.println(results.value, HEX);
    }
    irrecv.resume(); // resume receiver
  }
}

And there you have code for this particular IR remote set. It uses the NEC protocol. This software will echo back on the serial debug window the button you press on the remote. From there you can make it do whatever you want. It even filters out the annoying NEC "repeat code" when you hold down the button.

If you are using a different kit, or a different remote, the codes above may not be the same, or it may not be using the NEC protocol. The code above will spit out the received code.

You get output that looks like this:

Code:
Received Unknown: 4CB0FADD
Received Unknown: 4DB5DAAA
Received NEC: FFA25D 1
Received NEC: FF629D 2
Received NEC: FFE21D 3
Received NEC: FF22DD 4
Received NEC: FF02FD 5
Received NEC: FFC23D 6
Received NEC: FFE01F 7
Received NEC: FFA857 8
Received NEC: FF906F 9
Received NEC: FF6897 *
Received NEC: FF9867 0
Received NEC: FFB04F #
Received NEC: FF18E7 U
Received NEC: FF10EF L
Received NEC: FF38C7 O
Received NEC: FF5AA5 R
Received NEC: FF4AB5 D
Received Unknown: 24AE7D4E

Works well, but there are garbage "Unknown" results from the receiver picking up florescent lights or other stray IR sources. It prints out the HEX values for keys, so if your remote is slightly different, change the codes in the enum above to match the output you get. Just be sure to begin the HEX value with a "0x".

Identifying the protocol and tossing unknown results will give more reliable results. Protocols supported are:
NEC, SONY, RC5, RC6, PANASONIC, LG, JVC, AIWA_RC_T501, and WHYNTER. Check the IRRecvDump example in the Arduino IDE for examples on how to use them. I have NEC, so I'm using that.

So change the code in loop() to this:

C:
void loop() {
  if (irrecv.decode(&results)) {
    if (results.decode_type == NEC) {
      if (results.value != REPEAT) {
        Serial.print("Received NEC: ");
        Serial.println(nec_decoder(results.value));
      }
    }
    irrecv.resume(); // resume receiver
  }
}

Now the output is clean:
Code:
Received NEC: 1
Received NEC: 2
Received NEC: 3
Received NEC: 4
Received NEC: 5
Received NEC: 6
Received NEC: 7
Received NEC: 8
Received NEC: 9
Received NEC: *
Received NEC: 0
Received NEC: #
Received NEC: L
Received NEC: O
Received NEC: R
Received NEC: D

And there you have it, a simple IR remote for Teensy or Arduino.
 
Last edited:

Don't want to see this ad? Sign up for anRPF Premium Membershiptoday. Support the community. Stop the ads.

rworne

Member
Today is data entry day.

completed code for menu selection of train lines and stations:
C:
enum jr_lines
{
  JC = 0,
  JB,
  JT,
  JC2,
...
  JH,
  JO,
  MAX_JR_LINES
};


static const char JR_LINES[MAX_JR_LINES][32] = {"CHUO-RAPID LINE", "CHUO-SOBU LINE", "ITO LINE", "ITSUKAICHI LINE", "JOBAN LINE", "KEIHIN-TOHOKU LINE", "KEIYO LINE", "MUSASHINO LINE", "NAMBU LINE",
                                      "NEGISHI LINE", "OME LINE",  "SAIKYO LINE", "SHONAN-SHINJUKU LINE", "TAKASAKI LINE", "TOKAIDO MAIN LINE", "TSURUMI LINE", "YAMANOTE LINE",
                                      "YOKOHAMA LINE", "YOKOSUKA LINE"
                                     };

struct line_struct
{
  int count;
  int linenum;
  const char * line;
  const char ** stations;
};

static const char *JC_LINE[] = {"TOKYO", "KANDA", "OCHANOMIZU", "YOTSUYA", "SHINJUKU", "NAKANO", "KOENJI", "ASAGAYA", "OGIKUBO",
                                     "NISHI-OGIKUBO", "KICHIOJI",  "MITAKA", "MUSASHI-SAKAI", "HIGASHI-KOGANEI", "MUSASHI-KOGANEI", "KOKUBUNJI", "NISHI-KOKUBUNJI", "KUNITACHI",
                                     "TACHIKAWA", "HINO", "TOYODA", "HACHIOJI", "NISHI-HACHIOJI", "TAKAO"
                                    };

static const struct line_struct JCLINE = { 12, JC, JR_LINES[JC], JC_LINE};

static const char *JB_LINE[] = {"MITAKA", "KICHIOJI", "NISHI-OGIKUBO", "OGIKUBO", "ASAGAYA", "KOENJI", "NAKANO", "NIGASHI-NAKANO", "OKUBO",
                                     "SHINJUKU", "YOYOGI",  "SENDAGAYA", "SHINANOMACHI", "YOTSUYA", "ICHIGAYA", "IIDABASHI", "SUIDOBASHI", "OCHANOMIZU",
                                     "AKIHABARA", "ASAKUSABASHI", "RYOGOKU", "KINSHICHO", "KAMEIDO", "HIRAI", "SHIN-KOIWA", "KOIWA", "ICHIKAWA", "MOTO-YAWATA",
                                     "SHIMOSA-NAKAYAMA", "NISHI-FUNABASHI", "FUNABASHI", "HIGASHI-FUNABASHI", "TSUDANUMA", "MAKUHARIHONGO", "MAKUHARI", "SHIN-KEMIGAWA",
                                     "INAGE", "NISHI-CHIBA", "CHIBA"
                                    };

static const struct line_struct JBLINE = { 39, JB, JR_LINES[JB], JB_LINE};

static const char *JT_LINE[] = {"ITO", "USAMI", "AJIRO", "IZU-TAGA", "KINOMIYA", "ATAMI"};

static const struct line_struct JTLINE = { 6, JT, JR_LINES[JT], JT_LINE};

static const char *JC2_LINE[] = {"MUSASHI-ITSUKAICHI", "MUSASHI-MASUKO", "MUSASHI-HIKIDA", "AKIGAWA", "HIGASHI-AKIRU", "KUMAGAWA", "HAJIMA"};

static const struct line_struct JC2LINE = { 7, JC2, JR_LINES[JC2], JC2_LINE};

...
  
static const struct line_struct *JR_STATIONS[19] = {&JCLINE, &JBLINE, &JTLINE, &JC2LINE, &JJLINE, &JKLINE, &JELINE, &JMLINE, &JNLINE,
&JK2LINE, &JC3LINE, &JALINE, &JSLINE, &JULINE, &JT2LINE, &JILINE, &JYLINE, &JHLINE, &JOLINE};

And started entry of the train schedule for the Tokaido line heading outbound from Tokyo:
Code:
static const timetabletype timetabled[174] = {
  {5, 0, OUT_OF_SERVICE, MAX_LINES, UNDEF, MAX_VIA, MAX_DESTINATIONS, MAX_CARS, MAX_GREEN, MAX_DOORS, 0, MAX_TRACKS},
  {5, 0, OUT_OF_SERVICE, MAX_LINES, UNDEF, MAX_VIA, MAX_DESTINATIONS, MAX_CARS, MAX_GREEN, MAX_DOORS, 0, MAX_TRACKS},
  {5, 15, LOCAL2, UNDEF, UNDEF, UNDEF, ATAMI, CARS_15, NONRESERVED, DOORS_2, 0, TRACK_3},
  {5, 49, LOCAL2, UNDEF, UNDEF, UNDEF, ODAWARA, CARS_15, NONRESERVED, DOORS_2, 0, TRACK_3},
  {6, 8, LOCAL2, UNDEF, UNDEF, UNDEF, NUMAZU, CARS_15, NONRESERVED, DOORS_2, 0, TRACK_3}, // via tokyo
  {6, 27, LOCAL2, UNDEF, UNDEF, UNDEF, NUMAZU, CARS_15, NONRESERVED, DOORS_2, 0, TRACK_3}, // via shinjuku
  {6, 40, LOCAL2, UNDEF, UNDEF, UNDEF, ODAWARA, CARS_15, NONRESERVED, DOORS_2, 0, TRACK_3},
  {6, 56, LOCAL2, UNDEF, UNDEF, UNDEF, ATAMI, CARS_15, NONRESERVED, DOORS_2, 0, TRACK_3},

  {7, 5, LOCAL2, UNDEF, UNDEF, UNDEF, ATAMI, CARS_15, NONRESERVED, DOORS_2, 0, TRACK_3},
  {7, 13, LOCAL2, UNDEF, UNDEF, UNDEF, ODAWARA, CARS_15, NONRESERVED, DOORS_2, 0, TRACK_3}, // via shinjuku
  {7, 21, LOCAL2, UNDEF, UNDEF, UNDEF, ATAMI, CARS_15, NONRESERVED, DOORS_2, 0, TRACK_3},
  {7, 31, RAPID, UNDEF, UNDEF, UNDEF, HIRATSUKA, CARS_15, NONRESERVED, DOORS_2, 0, TRACK_3},
  {7, 36, LOCAL2, UNDEF, UNDEF, UNDEF, HIRATSUKA, CARS_15, NONRESERVED, DOORS_2, 22, TRACK_3},
  {7, 40, LOCAL2, UNDEF, UNDEF, UNDEF, ODAWARA, CARS_15, NONRESERVED, DOORS_2, 0, TRACK_3}, // via tokyo
  {7, 49, LOCAL2, UNDEF, UNDEF, UNDEF, ATAMI, CARS_15, NONRESERVED, DOORS_2, 2, TRACK_3},
  {7, 59, LOCAL2, UNDEF, UNDEF, UNDEF, ODAWARA, CARS_15, NONRESERVED, DOORS_2, 0, TRACK_3}, // via tokyo

...
};

Outbound trains are a lot more boring aside from the interesting destination names, especially with the station I selected.

Big problem coming up is how to get timetable data into the board. It sucks typing in 150+ entries for each direction for ONE STATION. There's 354 stations, and not all of them have only 2 tracks. That's a minimum of 106,200 line entries, double that if I want Saturday schedules, and triple for Sundays and holidays. It looks as if I'm going to have to scrape it, which means more work on the ESP8266 or I need to write something that downloads and formats it into something I can load in the Teensy. My search for machine-readable tables is coming up with zip. It looks like JR East is deliberately doing this, as they have an API access to pull this info on demand - for big bucks.
 
Last edited:

rworne

Member
Wow, what a colossal PITA that was. The JR East Page is set up such that you really cannot automate scraping all that well:
Screen Shot 2020-05-15 at 10.12.37 PM.png


General tools will trip up over the crazy table formatting, and it all requires some degree of manual intervention. So what you get after a lot of grief is something that *almost* works: A table with 70% of what you need, all that's missing is the hour :mad:.
Note: Green car info, the number of doors, track number, and the number of cars are not supplied by the JR site. Of course not. The train line is *implied" most of the time as well. Sometimes you get it from the info printed above or below the time, but mostly you need to click on the time to get a pop-up that gives you the info you need.

Screen Shot 2020-05-15 at 10.11.41 PM.png


So more manual editing, and bringing in Excel to process the data - it takes less than half the time, and the errors are down to pretty much zero. That I like.

Screen Shot 2020-05-15 at 10.10.55 PM.png


Copy and paste: straight in the code it goes and it compiles and runs without complaint.

I need to work on this further... getting the data in real time is the key. The real issue is picking the station. As JR doesn't want to make it easy, every station has an ID that doesn't make any sense aside from being a unique key. So this means generating code that will navigate to the data I want, then pulling it. And of course, any slight modification to the page layout will break everything.

I've also moved a bunch of data around to different files, to better organize things.

This was required because I finally have the data structures in place to hold display info for all the tracks on all the stations. For all but one line they are currently filled with copies of the data I already have, but all I need to do now is go in and change some numbers, a lot of numbers, and add any extra tracks I come across.

This is the 2nd largest personal software project I have undertaken, now totaling 90 source files and 55,202 lines of code.

Gave it a test run, along with a couple other minor mods to allow for the new coloring scheme found on the Chuō-line:
IMG_7473.jpg
IMG_7784.jpg


Looking pretty good so far.

Before anyone says anything, the schedule data it is using is from the Tokaido line, as there is no way the Chuō stops at these stations - but it does stop at three well-known geek locations: Akihabara, Nakano, and Mitaka. The first two homes to hobbyist and anime freaks, the latter home to the Ghibli museum.

Further update: Train schedule for Nakano Station is now added, so the Chuō line has realistic destination output.
 
Last edited:

rworne

Member
Well the menus are working, and any of the stations can be selected, but before that, I had to chase down an error with the pointers to the data structures. I think it's all worked out now.

So the next step: persistent memory.

When selecting options, such as a line, station, or the LED brightness, I do not want to have to select everything again each time I start up the program. I also do not want hard-coded values. So, write them to non-volatile memory, so whenever the board is powered up, it remembers what settings were there when it was powered off.

The code to do this with a Teensy is simple:
C:
#include <EEPROM.h>

#define INIT_ADDRESS 0x0000
#define LIN_ADDRESS  0x0004
#define STA_ADDRESS  0x0008
#define BRI_ADDRESS  0x00012

int station_index;
int line_index;
int led_brightness;

void setup() {
  Serial.begin(9600);
  if (EEPROM.read(INIT_ADDRESS) == 0xFF)
  {
    Serial.println("EEPROM not initialized");
    EEPROM.write(LIN_ADDRESS, 0);
    EEPROM.write(STA_ADDRESS, 0);
    EEPROM.write(BRI_ADDRESS, 25);
    EEPROM.write(INIT_ADDRESS, 1);
  }
  Serial.println("EEPROM initialized");
  led_brightness = EEPROM.read(BRI_ADDRESS);
  Serial.print("ledbrightness = ");
  Serial.println(led_brightness);
  line_index = EEPROM.read(LIN_ADDRESS);
  Serial.print("line_index = ");
  Serial.println(line_index);
  station_index = EEPROM.read(STA_ADDRESS);
  Serial.print("station_index = ");
  Serial.println(station_index);
  Serial.println("EEPROM read finished");
}

What this does:
Sets an address for three values, plus one extra for an initialization flag.
Creates 3 variables to hold these values.
Checks to see if the EEPROM was initialized - mine shows 0xFF. Yours may be different.
If it is uninitialized, but some safe values into the memory locations.

Then:
Read the three values and print them out.

Later in the code:
C:
void loop()
{
  ...
  if (button_pushed)
  {
    EEPROM.write(LIN_ADDRESS, line_index);
    EEPROM.write(STA_ADDRESS, station_index);
    EEPROM.write(BRI_ADDRESS, ledbrightness);
  }
  ...
}

Here we save the values to the EEPROM. Do not write continuously, as the EEPROM has a life of approximately 100,000 writes per address. Tie it to some function - a button press, so it only happens when you want it to.

And then you have power-up with saved states:
Code:
Starting ATOS Sketch
EEPROM initialized
ledbrightness = 25
line_index = 2
station_index = 12
Line/Station = CHUO-RAPID LINE/NAKANO
EEPROM read finished
WAV Trigger v1.32 
Number of tracks = 725
22:28:16
Starting Announcement
22:28:17
22:28:18

Anyhow, working on bugs and getting the menu, lines, stations and schedules working together. It's *almost* there.
Lines with no schedules downloaded display "not in service", and switching between lines switches schedules. Just need to do two things:
1. Add a count variable to the schedule structure, as the pointer to that object currently has no idea how many trains are due that day.
2. Add a parameter to switch directions, as an "up" schedule doesn't necessarily work well with a "down" matrix menu display, and vice-versa.

So Tokaido, Chuo-Rapid, and Yamanote lines are complete and working. 16 more lines to go.
 
Last edited:

rworne

Member
Threads revisited: Making highly accurate timers

Working with my blink thread, I had three timers running: a 700/200ms on/off blinky, a 500/500ms on/off blinky, and a 1000ms on/off blinky.

Like this:
C:
void blink_thread()
{
    while(1)
    {
          unsigned long temp = millis();
          blink_500_flag = (temp % 500);
          blink_1000_flag = ((temp % 2000) < 1000);
          blink_500_200_flag = ((temp % 700) < 500);
    }
}

Worked pretty well, except that every now and then, depending on what was updating and the CPU usage went a bit high, it was slightly "off" in timing. So slightly that you would not really notice it - until you do, then you cannot unsee it.

Time to try a similar, yet even more dangerous method of setting a timer - using interrupts.

The Teensy has an IntervalTimer, which is a highly accurate (for my use) timer that uses interrupts. I don't feel like pontificating on them at the moment, but it did look like it was going to solve my issue:
Code:
IntervalTimer myTimer;

volatile bool blink_500_200_flag = false;
volatile bool blink_1000_flag = false;
volatile bool blink_500_flag = false;
bool blink_500_200 = false;
bool blink_1000 = false;
bool blink_500 = false;

void blink_thread()
{
  unsigned long temp = millis();
  blink_500_flag = ((temp % 1000) < 500);
  blink_1000_flag = ((temp % 2000) < 1000);
  blink_500_200_flag = ((temp % 700) < 500);
}

void setup() {
  myTimer.begin(blink_thread, 100000); //fire off timer every 100ms
}

void loop() {
  noInterrupts();
  blink_500_200 = blink_500_200_flag;
  blink_1000 = blink_1000_flag;
  blink_500 = blink_500_flag;
  interrupts();

... //the rest of your stuff here
}

What does this give you? three simple boolean flags that are TRUE when the LEDs should be on, and FALSE when they should be off.

If you put
Code:
digitalWrite(ledPin, blink_500);
in the main loop(), you will get an LED blinking 1x/sec or 500ms on and 500ms off.

That's it, no more code to handle the LED, provided that's what you want it to do. If I put in:
C:
digitalWrite(ledPin, blink_500_200);
digitalWrite(ledPin2, blink_1000);
digitalWrite(ledPin3, blink_500);

We would have three LEDs blinking, all with accurate timing, and all in synch with each other, as the flags are all updated at once.

So remember how I said it was dangerous?
You need to ensure that whatever you do in your code for the interval timer is as short as possible. You should also not use any external libraries, and Serial.println is a definite no-no.

Additionally, you need to protect the variables you will be reading from being overwritten, even when you are currently reading them at the time. This is what the nointerrupts(), interrupts() functions come in. They protect the variables as they are being written. Forget them, and you may get glitching. forget one of them, and nothing will work.
 

rworne

Member
Possibly an update in a few more days - I had to take some time to address weird "exceptions" to the display rules.

The rules are generally simple:
Info 1 - Japanese
Info 2 - Japanese
Info 1 - English
Info 2 - English
In two rows.

Bottom row can be used for train approaching/passing warnings, the warnings blink 1000ms ON and 1000ms OFF.

Now I came up with these oddities:
Info lines with blinking text in them, with a 700ms on 200ms off duty cycle. How annoying.
New sizes for some of the fields, which required a lot of work obtaining new bitmaps.
Fields that changed sizes depending on what they were being shown.

And the real butt-kicker:
Displays that break the pattern above and have patterns like this:
Info 1 - Japanese - Announcement 2, Type
Info 2 - Japanese - Train Name
Info 3 - Japanese - Announcement 1
Info 2 - English - Train Name


After a lot of study, it turns out they were just being clever and were substituting different fields for certain trains (Narita Express, I'm looking at you)

and:
Info 1 - Japanese - Main Line, Type
Info 2 - Japanese - Line
Info 1 - English - Main Line, Type
Info 2 - English - Line
For others.

Identifying patterns is the key to good coding, and exceptions usually lead to bugs or display issues. I think I have a new system to work through this now.

I've also obtained 2020 Aluminum extrusion tubing and am expecting the screws and bolts over the weekend, along with the final display #6.

I also just ran into another issue: Scraping web pages. I was hoping to scrape some info from the JR site, but I've just been accessing it normally and doing the work by hand in Excel. Quite a bit though, as in a dozen times a night at most? So they noticed this, and while I am not doing anything improper, I'm greeted now by a captcha when looking at the timetables.
 
Last edited:

Don't want to see this ad? Sign up for anRPF Premium Membershiptoday. Support the community. Stop the ads.

rworne

Member
Screws and bolts arrived!

They are the wrong size! (my fault for not measuring it myself and going off a spec sheet).

Slight difference between M4 and M3 apparently :whistle:. Normally it doesn't matter, but the panels require an M3 threaded screw, the older panels must have used M4 screws.
 

rworne

Member
Now up to the final display size, blinking of elements enabled:

Ikebukuro Station - heading south, with Shonan-Shinjuku line to Odawara, followed by the Narita Express (NXP) to Narita Airport.

Did a bit more work on the timers - it turns out that millis() is not necessarily synched to the system time - you have about a 1/1000 chance of that happening. Blinkythings that operate on system time can blink slightly late or early if you need fractions of a second, because TimeLib only has 1 second resolution.

So I modified the TimeLib library to return the mills() count used to define the start of a second. This value is updated every time the time is set.

Subtract this from a millis() call in my code, and the resulting
Code:
(millis() - synchMillis) % 1000;
gives me the time difference from the last second in 1/1000 sec increments.
 
Last edited:

Inquisitor Peregrinus

Master Member
RPF PREMIUM MEMBER
You've done an amazing job here. I've sent a link to this thread to a programmer friend of mine who likes Japan, but -- I think -- has only been able to visit once, briefly. I'll agree with the sentiment that we need an electronics and coding subsection on the forum. There is so much out there to learn, but it's hard, without help or advice, to know where to focus. It's like looking at all of the Library of Congress and trying to figure out where you need to go without section signs. There are a lot of disciplines I know are useful to my work and hobbies, but I don't know the right questions to ask too much of the time.

You're really getting things dialed-in, though. I'm having flashbacks to when I lived over there. Ome Line, out Northwest of Tokyo. Change trains at Tachikawa if I was heading further in to the city proper. My own memory is packed with plenty of lines and stations just in the greater Tokyo area, never mind the rest of JR East! *lol*

The only thing that jumped out at me as odd was your calling this a "surreal" part of your reail experience over there. May I ask what prompted that word?
 

rworne

Member
The only thing that jumped out at me as odd was your calling this a "surreal" part of your reail experience over there. May I ask what prompted that word?

When you start to "get" what is going on at the train stations - the subtle psychological nudges, the sound effects, the weird blue LED lights at the end of the platforms - I noticed those lights on my last trip on the Chuo-line at the end of the platform. I thought it was a strange bug zapper or UV thing at first (those are actually there to prevent suicides).

direct.jpeg

Listen to any videos of stations and you hear the endless bird calls - they are there to guide the blind to the exits. Music is used to warn of departing trains, as they cause less anxiety than a buzzer or bell, prompting less running for the trains and injuries.

I've been there back in 1985, when I had to buy my tickets from a machine with no English capability, handing it to the guy rapidly clicking his hole-punch at the turnstiles to today where I just wave my phone at the gate as I enter the station.

I travel to Japan often, 1x or 2x a year (we had plans to leave for the summer trip tonight in fact - now canceled due to the pandemic) and I ride the lines frequently enough to enough destinations that I no longer stress out over how to get where I am going and spend more time observing what is going on around me. There's a lot to see in how they organize everything with visual cues - how can they move so many people so efficiently?

As a westerner, and one from a city that has crap for public transit, it's like another planet.

And you haven't lived until you push yourself onto an overloaded train that is jam-packed with passengers, the AC struggling with 80% + humidity and 95+ degree heat - the Yamanote at rush hour is a good example - they could not possibly get one more person on, can they? You could fall asleep standing and the press of bodies would allow you to sleep standing kind of crowded. Then you hit the next station and the people on the platform push in in mass, as everyone on the train is forced to simultaneously exhale to make room for another 30-50 bodies in the car.

SardinesCan-01.jpg


I love it there.
 

rworne

Member
Oh, JR East, why do you need to make my life so difficult?

So here's back at ya - a prototype web scraper to pull live schedules:

Screen Shot 2020-05-29 at 10.44.20 PM.png


Not bad for a night's work. ESP8266 Connects to the WiFi network, pulls the schedule down in a format similar to what the teensy needs:

Screen Shot 2020-05-29 at 10.51.31 PM.png


Compared to this:
Screen Shot 2020-05-29 at 10.32.13 PM.png

Needs a bit more work, but it'll do.

Of course the data is only a fraction of what I really need. Unlike just about everyone else who publishes schedules, JR seems to have contracted some company to publish theirs in a dead tree edition, so for online access everything you really need is hidden away behind a paywall. This company also has an iOS app to access their schedule database, and it also is lacking as much info as it can get away with. Yet Yahoo, Google, and I suppose Apple have access to all this. I could scrape both this page and generate a request for the next few trains from Yahoo, but I'm already having enough issues with memory constraints on the ESP8266 as it is.

I can get a bit more info processing the data above, but track numbers, # of cars, and the train #, are all deliberately hidden from view (some info is available from popup links). Reserved Seats can be gotten from the train name - special trains are typically reserved seats only. So I'll need to work on each direction of each line to get the "tokens" that identify the special trains, and when done, it can pull schedule info on demand.

What's left to do?

Get a the list of files from the server and build a lookup table to pull the appropriate webpage.

EDIT:
YES!!! I cracked the code. Need a lookup table for the stations, but the lines (and their schedule links) can now easily be predicted.
 
Last edited:

Don't want to see this ad? Sign up for anRPF Premium Membershiptoday. Support the community. Stop the ads.

rworne

Member
More work, more progress.

With a given station, I can bring up a table of schedules, identify what schedules they are, and extract the URLS for each:
Screen Shot 2020-05-30 at 8.11.43 PM.png


With this, I have the info I need to call up a schedule for a given station:
Station, Line, Direction, Weekday/Weekend Schedule.

EDIT:
Now working with multiple random stations. Fixed crashing bug when Shinkansen goes to the same station as the regular trains.
Fixed another bug where Tokyo station crashes the program. Max track schedules was = 20, Tokyo has: 22.
Discovered a way to get the host line (when another train line shares a track with another line)
Discovered a way to get VIA information
Now all that is left are holes in the lookup tables. Need to test it out on various lines and see what doesn't populate.
 
Last edited:

rworne

Member
The screws came in today, so I was able to attach the LED panels to the aluminium extrusions. It's officially a "sign" now, and looks way better than what I had before.

I was planning on hanging it up, but the sucker is heavy - all I know is I want it off my desk.

IMG_2717.jpg


Video of it in action:
 

rworne

Member
The best part of banging one's head against a wall - is when you stop.

The ESP8266 did an admirable job of scraping, but the limited resources prevented it from digging down more than two levels when scraping web pages. I happened to need three. It was also so slow, the website would often time out and download incomplete pages. I could still pursue the dynamic updates by throwing money at the problem and getting better hardware, until I noticed that JR hasn't changed their schedules very much over the past decade.

So I'll pull the info I need in chunks and save them to flash or SD card - depending on what fits. The internal clock and date will allow me to pull up the proper weekday/holiday schedule. All that's missing is track info (actually quite important) and some misc. info on the train itself (# doors, # cars).

So I went back and ported my web code to python, and had the Mac chew on it. Results are below:
Screen Shot 2020-06-05 at 10.35.07 PM.png

What you see on the bottom is the daily schedule information (some of it at least) for the JR East Tokaido Main Line, Fujisawa Station, heading north towards Tokyo on Track 1. Not seen above it is the schedule for the same station, heading in the opposite direction on Track 2.

It can pull all the schedule info off of 369 stations - and I'm missing a few that I need to fix as I see them crop up (as I see a bug in the data output right now): Field 4 is supposed to br the "UENOTOKYOLINE" and Field 5 is TOKAIDOSEN. This is one of those exception cases where a train line runs on a host line. So it needs a bit more work.

The bonus is I can have the code above generate the source code directly into my project, but I'd rather have it generate separate files for each track at the station. Each file generated is one track, so it's a simple matter of a global search and replace once I look it up.

To turn this mess:
Screen Shot 2020-06-05 at 10.57.45 PM.png


Into this:
IMG_1679.JPG


Nope, still haven't managed to get it off my desk.
 

rworne

Member
And the scraper is done. Needs a bit of variety to work on to stretch its legs, but it does the job. See the schedule in the previous post for the original data format:

Code:
Roberts-iMac:~ Robert$ python3 scraper.gyp

東海道本線
../2006/timetable/tt1361/1361020.html
../2006/timetable/tt1361/1361021.html
static const struct {
  int    count;
  timetabletype timetable[172];
} JT2_08_timetable_W_U = {
  172, {
{5,04,LOCAL2,UENOTOKYOSEN,TOKAIDOSEN,UNDEF,VIATOKYO,TAKASAKI,CARS_15,NON_RESERVED,DOORS_4,0,TRACK_1},
{5,31,LOCAL2,UENOTOKYOSEN,TOKAIDOSEN,UNDEF,VIATOKYO,MAEBASHI,CARS_15,NON_RESERVED,DOORS_4,0,TRACK_1},
{5,55,RAPID,UENOTOKYOSEN,TOKAIDOSEN,N_RABBIT,VIATOKYO,UTSUNOMIYA,CARS_15,RESERVED,DOORS_4,0,TRACK_1},
{6,05,RAPID,SHONANSEN,TOKAIDOSEN,UNDEF,VIASHINJUKU,TAKASAKI,CARS_15,NON_RESERVED,DOORS_4,0,TRACK_1},
{6,15,LOCAL2,UENOTOKYOSEN,TOKAIDOSEN,UNDEF,VIATOKYO,UTSUNOMIYA,CARS_15,NON_RESERVED,DOORS_4,0,TRACK_1},
{6,23,LOCAL2,UENOTOKYOSEN,TOKAIDOSEN,UNDEF,VIATOKYO,TAKASAKI,CARS_15,NON_RESERVED,DOORS_4,0,TRACK_1},
.
.
.
{22,29,LOCAL2,TOKAIDOSEN,TOKAIDOSEN,UNDEF,UNDEF,TOKYO,CARS_15,NON_RESERVED,DOORS_4,0,TRACK_1},
{22,37,LOCAL2,TOKAIDOSEN,TOKAIDOSEN,UNDEF,UNDEF,TOKYO,CARS_15,NON_RESERVED,DOORS_4,0,TRACK_1},
{22,40,RAPID,SHONANSEN,TOKAIDOSEN,UNDEF,VIASHINJUKU,TAKASAKI,CARS_15,NON_RESERVED,DOORS_4,0,TRACK_1},
{22,50,LOCAL2,TOKAIDOSEN,TOKAIDOSEN,UNDEF,UNDEF,TOKYO,CARS_15,NON_RESERVED,DOORS_4,0,TRACK_1},
{23,03,LOCAL2,TOKAIDOSEN,TOKAIDOSEN,UNDEF,UNDEF,TOKYO,CARS_15,NON_RESERVED,DOORS_4,0,TRACK_1},
{23,18,LOCAL2,TOKAIDOSEN,TOKAIDOSEN,UNDEF,UNDEF,SHINAGAWA,CARS_15,NON_RESERVED,DOORS_4,0,TRACK_1},
{23,30,LOCAL2,TOKAIDOSEN,TOKAIDOSEN,UNDEF,UNDEF,SHINAGAWA,CARS_15,NON_RESERVED,DOORS_4,0,TRACK_1},
{23,43,LOCAL2,TOKAIDOSEN,TOKAIDOSEN,UNDEF,UNDEF,SHINAGAWA,CARS_15,NON_RESERVED,DOORS_4,0,TRACK_1},
  }
};

static const struct {
  int    count;
  timetabletype timetable[184];
} JT2_08_timetable_H_U = {
  184, {
{5,04,LOCAL2,UENOTOKYOSEN,TOKAIDOSEN,UNDEF,VIATOKYO,TAKASAKI,CARS_15,NON_RESERVED,DOORS_4,0,TRACK_1},
{5,31,LOCAL2,UENOTOKYOSEN,TOKAIDOSEN,UNDEF,VIATOKYO,MAEBASHI,CARS_15,NON_RESERVED,DOORS_4,0,TRACK_1},
{5,55,RAPID,UENOTOKYOSEN,TOKAIDOSEN,N_RABBIT,VIATOKYO,UTSUNOMIYA,CARS_15,RESERVED,DOORS_4,0,TRACK_1},
{6,05,LOCAL2,TOKAIDOSEN,TOKAIDOSEN,UNDEF,UNDEF,OSAKA,CARS_15,NON_RESERVED,DOORS_4,0,TRACK_1},
.
.
.
{23,30,LOCAL2,TOKAIDOSEN,TOKAIDOSEN,UNDEF,UNDEF,SHINAGAWA,CARS_15,NON_RESERVED,DOORS_4,0,TRACK_1},
{23,43,LOCAL2,TOKAIDOSEN,TOKAIDOSEN,UNDEF,UNDEF,SHINAGAWA,CARS_15,NON_RESERVED,DOORS_4,0,TRACK_1},
  }
};

Every station I put into it, it grabs the schedule for weekday and holiday trains for every train line that goes to that station. It looks up the station code and direction, generates a variable name from it, then creates a C structure that can be copied and pasted into my code. It also counts the number of trains arriving, so the array is properly accessible.

Now, what to do with it. I can just download the data as it is today, since I obviously cannot catch the trains in Tokyo from the west coast of the USA, so who cares if it ever gets out of date? But I still want to see if I can somehow manage to pull the data live... But that's a battle for another day I guess.
 

Don't want to see this ad? Sign up for anRPF Premium Membershiptoday. Support the community. Stop the ads.

Don't want to see this ad? Sign up for anRPF Premium Membershiptoday. Support the community. Stop the ads.

Top