JARVIS ver 3.0

Looking good!


I had attempted (with member TheRealStark) an HUD display before.. (with real time info fed back from the Arduino)..

This was obviously YEARS ago.. and my initial approach was to use Adobe Flash as the webcam/gui overlay visuals..

it worked great as a concept run a PC.. but I could never find a hardware solution that could stream the live webcam feed -AND- run the Adobe Flash Player plug-in..

If I recall..I think we were looking the PCDuino? and eventually the Raspberry Pi..

both of which (as far as I ever learned) could NOT get the flashplayer running in their native set-ups.
 
Looking good!


I had attempted (with member TheRealStark) an HUD display before.. (with real time info fed back from the Arduino)..

This was obviously YEARS ago.. and my initial approach was to use Adobe Flash as the webcam/gui overlay visuals..

it worked great as a concept run a PC.. but I could never find a hardware solution that could stream the live webcam feed -AND- run the Adobe Flash Player plug-in..

If I recall..I think we were looking the PCDuino? and eventually the Raspberry Pi..

both of which (as far as I ever learned) could NOT get the flashplayer running in their native set-ups.

This particular build is actually running on a Raspi Pi 3. The Processing program now allows for decent hardware accelerated video imaging. Right now it runs at a smooth 30 fps 800x480 resolution but that's with the first version raspberry pi camera. At some point, I plan on getting the second version of the camera so I can get higher frame rates. I've taken a sort of break from it so I don't get burnt out too quickly. I plan on coming back to it in the few weeks as I was making good progress on an actual working hud. All in all at a steady a 30 fps at less than 5% cpu is really really good and so far it been pretty cheap to build with all things considered.

One thing I did want to add is that if can ever get my hands on to that Google voice recognition hat for the pi, it would make Jarvis even more real and possible.
 
Do a search for the JASPER project. :)

Open Source too.. (last I recall)

Good idea on coming back to the project to avoid burn out! (my problem is... I dont come back some times!.. or its months before I'm back on a side project and then only for another couple weeks!) LOL

I have not messed with any processing projects before.. so I''m not clear how you can use/make graphics much.

While Flash is more or less dead tech... (at least on the web it should be).. in a closed format or a gui/menu type projects it was (is?) a great way to make some really cool stuff/interfaces..etc...etc.

which is why it lent itself great to that HUD project.. (simple overlay of real-time stream webcam)

Even with the Pi3 being more powerful (I actually have two sitting right here!) :).. I still dont believe its possible to get it running flash player in any sort of browser still.

Good luck!
 
The processing program is super simple to figure out especially if you are familiar with the Arduino. It only took me a couple of hours to install it and write about half of the final version. I'm for sure going to look into the Jasper project :). As far as flash goes on the pi, I did find some info on how to load it on there. This particular example uses the first gen PI and not the PI 3 and I would interested to see how it performs since it says that on the first gen Pi, it is slow (but that wasn't on the Raspian Distro). I did pick up an Asus tinkerboard when I was at Microcenter during my last Con and have found it it to be quite a bit faster compared to the pi. The software on that is good, it just needs to be ironed out some and I'm hoping it will be compatible with the PI camera. At that point I would ditch the pi for good.
 
Let's say you were able to find maybe a curved, more high definition screen. What if you used a pi camera module, and used that to create a heads up display inside the helmet, superimposing details onto the screen.
 
Let's say you were able to find maybe a curved, more high definition screen. What if you used a pi camera module, and used that to create a heads up display inside the helmet, superimposing details onto the screen.



??

umm.... thats what we are talking about here!?

*the curved display is really whatever the end user chooses to use IMHO.. (whatever fits the project)


We -are- discussin the topions, whats been working, whats been tried before...etc


I read RPi/Chromium supports flash player!

I dont have a Pi Cam to test with though...
devildog12

Do you have a Pi3 set-up with web cam? (to do a quick test?)

Do any of these links work for you?

http://dmstudios.net/misc/HUD_test/

http://dmstudios.net/misc/HUD_test/HUD_overlay_demo1.html

http://dmstudios.net/misc/HUD_test/HUD_overlay_demo2.html


(quick effect test or the 'damaged mode' of the HUD)

http://dmstudios.net/misc/HUD_test/TV_test1.html


While flash is mostly dead for the web (or should be).. I think this is still a great use for it.

If this works on the Pi.. I think we maybe even more options. Since it can have a web server as well... we can run Python scripts and communicate via serial/comm to commiunicate to the HUD from other auxilaryu pieces (uni-beam, repulsors.. armor updates/status..etc)
 
Last edited by a moderator:
@xl97 I'll try running those links later tonite. Writing a Python script to interact with the camera and other serial devices is a pain because Python doesn't like to do dynamic text overlays over top the camera. I don't know it has something to do with Python not being fast enough to lay text or images over top the frame buffer. I was going to do everything in Python but the Processing app was much simpler.

@MJ43 A curved display would be crazy expensive to add in the helmet. However I do have video glasses with a resolution of 240x320 per eye and everything comes up super small even when it coded to match that display resolution. My plan is to either purchase higher def video glasses or do something along the same lines a google glass. I'm also kind of hoping the Microsoft Hololens won't be ridiculously priced when it hits the consumer market because would try those with a windows-based sbc (small board computer).
 
Okay, what if you were to wire up the screen yourself, maybe try something like the google glass where you use a projector with a prism to direct it into your eye, or, perhaps something with graphene, which could work extremely well. With the google glass option, you may not even need a camera, as you are just superimposing things onto reality.
 
@xl97 I'll try running those links later tonite. Writing a Python script to interact with the camera and other serial devices is a pain because Python doesn't like to do dynamic text overlays over top the camera. I don't know it has something to do with Python not being fast enough to lay text or images over top the frame buffer. I was going to do everything in Python but the Processing app was much simpler.

@MJ43 A curved display would be crazy expensive to add in the helmet. However I do have video glasses with a resolution of 240x320 per eye and everything comes up super small even when it coded to match that display resolution. My plan is to either purchase higher def video glasses or do something along the same lines a google glass. I'm also kind of hoping the Microsoft Hololens won't be ridiculously priced when it hits the consumer market because would try those with a windows-based sbc (small board computer).


OH.. sorry if I wasnt clear.

The python would just be to interact with the Arduino/device side of things.. (part of the device feedback system).... (ie: serial comm communication to pass on this data to the Flash app/overlay)..

I have built a few 'apps' that use Flash as a front end/GUI.. and then used some PHP to pass on this data to connected Arduino..

the problem with using PHP (at least on a PC) is that communication is only one way... (no feedback from the Arduino only too)... but we can use Python.

Actually (DUH).. since the Pi is a PC but in fact "Linux".. maybe we can still use PHP? (which I'm much better at than Pyhton!) LOL


Let me know if you get a chance to boot up a Pi3 with a PiCam attached and can visit those links..

It 'should' (in theory) ask to activate your web cam... then display a live stream, with a generic overlay. (just a test for usability)..

If this works... we can work together (if you like) to hash out some features/animations...etc.. to be displayed.
 
OH.. sorry if I wasnt clear.

The python would just be to interact with the Arduino/device side of things.. (part of the device feedback system).... (ie: serial comm communication to pass on this data to the Flash app/overlay)..

I have built a few 'apps' that use Flash as a front end/GUI.. and then used some PHP to pass on this data to connected Arduino..

the problem with using PHP (at least on a PC) is that communication is only one way... (no feedback from the Arduino only too)... but we can use Python.

Actually (DUH).. since the Pi is a PC but in fact "Linux".. maybe we can still use PHP? (which I'm much better at than Pyhton!) LOL


Let me know if you get a chance to boot up a Pi3 with a PiCam attached and can visit those links..

It 'should' (in theory) ask to activate your web cam... then display a live stream, with a generic overlay. (just a test for usability)..

If this works... we can work together (if you like) to hash out some features/animations...etc.. to be displayed.

So I tried it with my pi cam and it doesn't look like it works. It asks to use Flash then a blank screen. I tried on my main computer that has a webcam attached and it did the same thing. Maybe the flash code needs to be updated? As far as the python code goes, no worries. You can actually pull serial data in from the arduino while also using the webcam interface. I just haven't got around to actually coding that in. You can install Firmata on the arduino and use it that way or you can write a simple serial function to export the data from the arduino to the Processing "sketch".
 
Thanks for the test.


I'll go back and re-check things... (I think I may have a netbook upstairs with a built in webcam?)

Asks to use flash... do you ever get a prompt to use/activate the cam?

Have you refreshed? (I thought I read somewhere that its no enabled by default....etc and doing something and re-loading the page made it show up for/work)

*I'll have to go back a read about what was said about using Chromium..etc


This link was just a simple flash movie:

http://dmstudios.net/misc/HUD_test/TV_test1.html


(no web cam involved)

that would/could be a basic Flash test... that one did not work as well?

(bummer)
 
Thanks for the test.


I'll go back and re-check things... (I think I may have a netbook upstairs with a built in webcam?)

Asks to use flash... do you ever get a prompt to use/activate the cam?

Have you refreshed? (I thought I read somewhere that its no enabled by default....etc and doing something and re-loading the page made it show up for/work)

*I'll have to go back a read about what was said about using Chromium..etc


This link was just a simple flash movie:

http://dmstudios.net/misc/HUD_test/TV_test1.html


(no web cam involved)

that would/could be a basic Flash test... that one did not work as well?

(bummer)

The last one works :) Sorry I forgot to check that one, however I did check the other 3 though. There wasn't the usual flash prompt asking for camera permissions on the other 3 on the Pi or on my main machine. I did go into the settings for flash and look to see if the camera was even available and it wasn't. I'm wondering if I hooked up my regular webcam to see if was available. I also noticed that chromium (when running Flash) used about 30% of the CPU.
 
RE: 30% of CPU use..

THAT I can believe...

Was a bit of a resource intensive hog.. (especially when animations were running)... so that 'HUD damage mode' test I'm sure is pretty intense.


Thanks for the tests!!..


So we have confirmed, that FLASH does in fact work on the Pi now! (good)..

I'll have to either try out my old netbook upstairs.. or dig out my old webcam.. and give it a shot to see about the camera stuff again. (I dont recall anything security related to the webcam in the past? so I dont think it was removed/disabled or anything.)

It requires the user interaction to 'approve' that flash is going to use the webcam.. (it wont do it by default, without consent) *(which may be an issue now that I think about it!) LOL

but I'll go back and get out the flash movie see if I can figure anything out.. or test on a PC with webcam..
 
I'm gonna try getting Plug 'N Play mic from Amazon this weekend and try the Jasper Project out. I was also going to check IBM Watson for a real "Natural-Language" Ul. I think that paired with all my other bells and whistle in the helmet and forearm would be mind blowing.
 
I have been toying with the idea to give the Jasper project a 'go' myself... (so I'll be interested to see what you attempt/get going)..
 
So I took a deep dive into the Jasper project and it looks like there is a lot of confusion between the actual image and the "working image". Also it looks like it won't work with the Pi 3. So I decided to take a look into new google version of voice recognition. The magazine MagPi #57 had a special voice recognition hat that allowed for home automation, and after some digging on the web I found that I can actually use that image and enable the onboard audio on the Pi and make it work without the special hat that came with the magazine. Hopefully after tinkering some and seeing how that works hopefully I can have semi-working prototype by the weekend. **Fingers Crossed** Basically what that would mean is that Google servers are handling Jarvis voice recognition with the automation of the electronics handled locally strictly on the Pi.
 
After about an hour of tinkering with Pi and Google, I have a working voice recognition, now to just integrate everything with the camera HUD interface. Hopefully I can pull this off ;)
 
Looking really good! Excited to see the final set up.

I recently incorporated voice recognition into my Ironman suit also. I used an easyVR and arduino though, but it was good enough for my purposes. I have a friend who does voice acting and impersonations, so I got him in the studio with a mic and recorded custom JARVIS files.

I have been looking into adding muscle sensors to my forearm set up also. I would love to see more about your experience working with the EMG sensors.
 
Looking really good! Excited to see the final set up.

I recently incorporated voice recognition into my Ironman suit also. I used an easyVR and arduino though, but it was good enough for my purposes. I have a friend who does voice acting and impersonations, so I got him in the studio with a mic and recorded custom JARVIS files.

I have been looking into adding muscle sensors to my forearm set up also. I would love to see more about your experience working with the EMG sensors.

Thanks! Its going to be awhile I believe for the final version mainly for the fact I'm offloading as much as I can onto the Raspberry PI as far as sensors and such. The EMG sensor is a great piece of cosplay tech. I honestly think its underused and under valued.You can see my "prototype" replusor tech from a couple years ago here. As far as the Jarvis sound files, that pretty awesome. I mainly just clipped every Jarvis soundbyte I could find from the movies all the way back to Iron Man 1. I've been thinking about doing a FRIDAY setup once this finally done. That way when I release everything, you can use either one.
 
This thread is more than 5 years old.

Your message may be considered spam for the following reasons:

  1. This thread hasn't been active in some time. A new post in this thread might not contribute constructively to this discussion after so long.
If you wish to reply despite these issues, check the box below before replying.
Be aware that malicious compliance may result in more severe penalties.
Back
Top