New TurtleCam Server

[RJM62]

My turtle site has been getting a lot more traffic and has outgrown the windows-based freeware that had been handling the video feed. It had been running reliably on an old, tiny Acer notebook for about five years, so I got a lot of mileage out of that setup.

The new server is also based on an old machine, in this case a laptop that I retired about a year or two ago. I backed up the Windows system onto its second hard drive, installed CentOS 7 on it, and am using FFmpeg and FFserver to capture the webcam video and convert it to an flv stream.

I then created a reverse proxy on the Apache server that serves the site, and installed mod_flvx and a handler so Apache would handle the flv stream.

So far it seems to be working, although I need to tweak it for quality and so forth. I also plan to use a better streaming format eventually. I just haven't decided which one yet.

You can (hopefully) see it working at

http://www.myturtlecam.com/liveFeed/index.php

I've also re-written the site recently on the advice of a few teachers who emailed me and told me that they used it in their classes, but that the reading level was too high. So I rewrote it to be more kid-friendly.

Rich

 

[Everskyward]

Interesting. I watched for a minute and it looked like they were mating. Or is that how turtles do it. I have no idea.

 

[RJM62]

Interesting. I watched for a minute and it looked like they were mating. Or is that how turtles do it. I have no idea.

No, not in this case. They're both females. At least that's what the vet said.

Rich

 

[EdFred]

Never loaded for me. Win7x64 ultimate FireFox 35.0

 

[Norman]

Worked for me. Win 7x32. Firefox 35 something or other.

 

[EdFred]

Going back to it, it popped right up.

 

[RJM62]

Yep, I still need to tweak some things, apparently.

Rich

 

[steingar]

Came up, but I didn't see any turtles. I suppose I could install a python-cam in my home office, but they really don't do all that much apart from occasionally eating vermin.

 

[flyingron]

What are the varieties? I didn't see that listed. I'm sitting here in my office next to a Florida River Cooter (we had a red-eared slider but she passed years ago). Of course, it's a lot less animated (though it still likes music, they share the room with a grand piano) then the parrot that is currently off his perch and on his favorite spot my right shoulder ...

 

[kayoh190]

Works great for me. I run Safari on a Mac.

Pretty cool seeing the turtles swim around!

 

[RJM62]

Came up, but I didn't see any turtles. I suppose I could install a python-cam in my home office, but they really don't do all that much apart from occasionally eating vermin.

It'll be up and down for a while. I've been tweaking settings this morning. I also moved the physical machine and shut it down to do so -- stupidly forgetting that it was a laptop and therefore had its own battery that could have powered it during the move.

Tweaking this setup is going to be a fairly tedious process because in addition to the normal considerations of balancing quality, bandwidth, and so forth, there are a bazillion or so quirks to FFmpeg itself; and despite its ubiquity, FFmpeg's documentation is... let's just say "less-than-spectacular." There's the man page, of course, and there's official documentation on their site; but both are rather vague and sometimes outdated.

There's also a patchwork FFmpeg "community" that's willing to share what works well for them, but that doesn't necessarily mean it will work well for anyone else. The software is extremely finicky about the way that different configuration settings work with each other, making the process of optimizing a particular setup a trial-and-error sort of process.

But that's fine. Once I'm done I'll have an extra skill. That's never a bad thing.

Rich

 

[RJM62]

What are the varieties? I didn't see that listed. I'm sitting here in my office next to a Florida River Cooter (we had a red-eared slider but she passed years ago). Of course, it's a lot less animated (though it still likes music, they share the room with a grand piano) then the parrot that is currently off his perch and on his favorite spot my right shoulder ...

Southern Painted Turtles (Chrysemys picta dorsalis). That's mentioned elsewhere on the site. They're my personal favorite and my recommendation for new turtle keepers because they are hardy, have great personalities, are very active and animated even as adults, and are among the smaller aquatic turtles (which means you don't need quite so big a tank, filter, etc.).

I suspect the reason Southern Painted Turtles are not very popular as "pets" is because of a federal law forbidding the sale of "pet" turtles with carapace lengths of less than 4 inches. That makes it impractical for pet shops to sell physically smaller species. There are ways around it (mainly, by declaring the turtles a "hobby" rather than "pets" and dealing directly with a breeder), but most pet shops aren't going to bother with that.

Rich

EDIT: I've added the turtle specie to the habitat description on the video page, Ron. Thanks. That was a good idea. There are a lot of visitors who land on that page without having visited any other pages on the site.

 

[steingar]

Now there are turtles. Makes me nostalgic, I used to have them as kids. Sort've like tropical fish, except far more interactive and entertaining.

 

[RJM62]

Now there are turtles. Makes me nostalgic, I used to have them as kids. Sort've like tropical fish, except far more interactive and entertaining.

I agree. But they're also a commitment.

My 10-year-old nephew recently told me that he wants to get into the hobby. I tried my best to discourage him -- kind of like a rabbi is supposed to discourage prospective converts -- to make sure that he understands what he's getting into.

When I told him that if he gets hatchlings and takes care of them, it's entirely possible that he'll still have those turtles when he's a dad or a granddad, that made him think for a while, but he decided that that was okay. He read my entire turtle site (I actually plan to pay him for the proofreading and reading-level check), and nothing there changed his mind, either. He really wants to keep turtles.

The final test will come in February when my brother's bringing him up here. He agrees that the little guy needs to understand what he's getting into. Among other things, I plan to have my nephew change the media in the filter. It's a huge filter, so even the physical effort of moving it will be considerable for a little guy. If he can deal with that plus the surprises that await him inside, then I'll know he's ready and will help him set up a habitat.

On a technical note, I've set up an alternate video stream using FFserver rather than Apache on the Web sever to serve the final stream (while continuing to do the transcoding in my office, since that machine has no other duties except the video streaming). I figure FFserver is actually designed to serve video streams, so it may do it better / more efficiently than Apache with mod_flvx. To determine whether that's the case, I'm doing some tests that will require me to edit the config files and restart both FFmpeg and FFserver from time to time, meaning that there will be outages. My apologies.

Rich

 

[RotorDude]

I agree. But they're also a commitment.

My 10-year-old nephew recently told me that he wants to get into the hobby. I tried my best to discourage him -- kind of like a rabbi is supposed to discourage prospective converts -- to make sure that he understands what he's getting into.

When I told him that if he gets hatchlings and takes care of them, it's entirely possible that he'll still have those turtles when he's a dad or a granddad, that made him think for a while, but he decided that that was okay. He read my entire turtle site (I actually plan to pay him for the proofreading and reading-level check), and nothing there changed his mind, either. He really wants to keep turtles.

The final test will come in February when my brother's bringing him up here. He agrees that the little guy needs to understand what he's getting into. Among other things, I plan to have my nephew change the media in the filter. It's a huge filter, so even the physical effort of moving it will be considerable for a little guy. If he can deal with that plus the surprises that await him inside, then I'll know he's ready and will help him set up a habitat.

On a technical note, I've set up an alternate video stream using FFserver rather than Apache on the Web sever to serve the final stream (while continuing to do the transcoding in my office, since that machine has no other duties except the video streaming). I figure FFserver is actually designed to serve video streams, so it may do it better / more efficiently than Apache with mod_flvx. To determine whether that's the case, I'm doing some tests that will require me to edit the config files and restart both FFmpeg and FFserver from time to time, meaning that there will be outages. My apologies.

Rich

If this is going to school kids, you might want to fix this.

 

[steingar]

Chelonians are cool. They evolved antecedant to the dinosaurs and survived them. Who would've ever thought an armored reptile would be semiaquatic?

 

[RJM62]

If this is going to school kids, you might want to fix this.

You know, I've heard much smarter people than myself argue both sides of that question. I think I'll take your advice, however, because it's in common usage; and even were it wrong, it wouldn't insult anyone except a few eggheads.

Rich

 

[RotorDude]

You know, I've heard much smarter people than myself argue both sides of that question. I think I'll take your advice, however, because it's in common usage; and even were it wrong, it wouldn't insult anyone except a few eggheads.

Rich

Sounds good. I think with Paul Brians watching your six, you'd be safe. :wink2:

 

[RJM62]

I have the video feed running on the alternate stream now, using ffmpeg to push the stream on the sending side and ffserver to receive it upstream and serve it to the visitors. The arrangement has its plusses and minuses.

On the plus side, the picture quality is a bit better using ffserver. It also has a lot less lag time and handles multiple requests much better than Apache with mod_flvx does. Apache balked a few times when there were only 20 simultaneous connections, but ffserver doesn't even shrug serving twice that many.

On the minus side, ffserver uses more resources than Apache. It's not a major big deal because I have sufficient resources and could buy more if I needed them. It just surprised me. It ranks right up there with spamd, mysql, and exim in terms of appetite.

One thing I'm having a tough time with is getting the connection stable when I use ffserver on both ends (in my office to serve the files via HTTP when requested by the Web server, and on the Web server to serve the files to visitors). Sometimes ffserver on the server in my office will accept the connection, and sometimes it won't. It doesn't seem to be resource, bandwidth, or DNS-related. When I try to access the stream using a browser or VLC, it's there; but when the upstream server tries to grab it, it's hit or miss. I frankly don't know where the problem is.

It's wonderfully stable in the other direction (using ffmpeg locally and pushing the connection to ffserver on the Chicago server), which is how I'm doing it now. But that requires a constant data stream, which consumes bandwidth even when no one's viewing the video. I haven't yet figured out an elegant way to remotely start the capture / transcode / push from my office only when requested by the upstream server.

Ffserver does have a status page that tracks connections, so maybe I can tap into that output. Or I suppose I can look at the source code and insert a command to remotely launch ffmpeg upon an HTTP request, and then recompile it with that feature.

Another option would be to just cron ffmpeg off at night since there's not much to see, anyway. I can write a little script to cron ffmpeg on and off and at the same time send a message to the upstream server to replace the player with a message saying, "The turtles are sleeping," or some such thing at night. On average, I only have 20 or so connections on average between 11:00 p.m. and 7:00 a.m. ET, so it seems a bit wasteful to burn up bandwidth that way when there's nothing to see.

Then again, I have a business Internet account with an 750 GB soft cap, and if I ever exceeded it all that would happen would be that I'd get a phone call from Pete asking me how the hell I'd burned up 750 GB; so maybe I shouldn't worry about it.

Rich

 

[RJM62]

If anyone's interested (for purely geekly reasons), here are the settings I'm using:

ffserver (upstream / Web server):

Port 8090
BindAddress 0.0.0.0
MaxHTTPConnections 200
MaxClients 100
MaxBandwidth 10000
CustomLog -

<feed turtle.ffm>
        File /tmp/turtle.ffm
        FileMaxSize 5M
        ACL allow 127.0.0.1
        ACL allow [my office ip]
</Feed>

<Stream turtle.flv>
        Feed turtle.ffm
        Format flv
        VideoCodec flv
        VideoFrameRate 15 <- this is ignored for some reason
        VideoBufferSize 800
        VideoBitRate 500
        VideoQMin 10
        VideoQMax 20
        VideoSize 640x480
        PreRoll 0
        NoAudio
        AVOptionVideo flags +global_header
        AVOptionAudio flags +global_header
</Stream>

<Stream stat.html>
        Format status
        ACL allow localhost
        ACL allow [my office ip]
</Stream>

<Redirect index.html>
        URL http://www.myturtlecam.com/
</Redirect>

And the ffmpeg command on my local machine:

ffmpeg -f video4linux2 -i /dev/video0 -r 15 -s 640x480 http://[Chicago server IP]:8090/turtle.ffm

The framerate directive is completely ignored by the Logitech C270 camera, but respected by a crappy old one I have laying around in my office. The C270 also doesn't properly scale its image when put into HD mode with the Linux UVC driver. If I try to put it into any aspect ratio other than 4:3, it distorts the output. Possibly there's a firmware update or a hacked driver that would fix those problems. But those things are for another day.

Rich

 

[Henning]

Considering the Porta potty thread, I was expecting something very different...

 

[RJM62]

Well, I went into "Obsession Mode" and solved my connection problem. It turned out that the idiot who was working on it this afternoon -- that would be me -- committed a typo when opening a firewall port on the Chicago server. Doh!

The reason I thought it was an intermittent issue was because one of my test pages was still going through the reverse proxy and the other one wasn't. Again I say, Doh!

Now that I've cleared that up, I'm running both ffmpeg and ffserver on my office machine to capture and encode the webcam stream, as well as serve it to the one allowed client: the Chicago server, which also uses ffserver to serve the feed to the WWW because it does that better than Apache does.

The drawback is a several-second delay before the video starts that I think is a DNS issue that I can fix. But the advantage is that now the feed only draws data when someone's actually watching the video. That should save me about 100 - 150 GB / month of bandwidth on both my office account and the Chicago account.

Life is good.

Rich

 

[flyingron]

Southern Painted Turtles (Chrysemys picta dorsalis). That's mentioned elsewhere on the site. They're my personal favorite and my recommendation for new turtle keepers because they are hardy, have great personalities, are very active and animated even as adults, and are among the smaller aquatic turtles (which means you don't need quite so big a tank, filter, etc.).

Cool. Yeah, ours (the cooter) is at least twelve years old (it was a classroom pet my wife inherited from another science teacher). She's already in a 100 gallon tank and we're probably going to have to make a bigger one (won't be glass next time) in a couple of years.

Some of us remember the little tank with the small turtles at the dime store.

 

[RJM62]

Cool. Yeah, ours (the cooter) is at least twelve years old (it was a classroom pet my wife inherited from another science teacher). She's already in a 100 gallon tank and we're probably going to have to make a bigger one (won't be glass next time) in a couple of years.

Some of us remember the little tank with the small turtles at the dime store.

Yeah... unfortunately, they still sell hatchling red-eared sliders in Chinese neighborhoods in NYC on the sidewalk and at fairs, which is why so many of the lakes and ponds downstate have been become completely full of abandoned RES. People don't understand the size those little turtles grow to and the commitment they represent.

I was considering building a pond up here for some rescues, but DEC doesn't think my proposed location is especially suitable (more because of climate than any other reason).

Rich

 

[steingar]

Cool. Yeah, ours (the cooter) is at least twelve years old (it was a classroom pet my wife inherited from another science teacher). She's already in a 100 gallon tank and we're probably going to have to make a bigger one (won't be glass next time) in a couple of years.

Some of us remember the little tank with the small turtles at the dime store.

I had a pet tortoise when I was in graduate school. Had Grundy the tortoise all the way through tenure. She wound up catching some tortoise disease and died quietly one cold winter. Afterwards Mrs. Steingar informed me that our dining room was to become a chelonian free zone.

I miss my tortoise. When I would cook I would always chop up some veggies extra fine for Grundy. That was really cool.

 

[Seanaldinho]

My family has a florida cooter that was found in a parking lot and was brought home when it was about the size of a silver dollar. 3 aquariums later its the size of a small dinner plate and is expecting a new tank in the near future.

Dumb as he is, he's fun to watch.

 

[flyingron]

Margy has allocated room in the pool room of the airport house for the turtle habitat. I suggested we just let him live in the pool, but she won't go for that.

We've got cooters in the wild in the lake here. They hang out around the fuel dock at the marina hoping some kids will drop some fishfood over the edge.

 

[RJM62]

Okay, I changed the streaming setup again. Pulling the content was spawning too many connections and ffmpeg children. So I changed it back to a push; but to reduce the bandwidth, I wrote a few scripts to remotely start the stream only when the page is loaded. It's not perfect yet, but it works.

I couldn't get ffmeg to daemonize (apparently that's a not-uncommon problem), so I decided to accept less-than-perfect for the time being and write a little shell script to check whether the process was running, and if not, to start it.

Also to save bandwidth, I inserted a 10-minute duration on the ffmpeg process. This is one of the imperfect parts. If a user opens the page before an existing ffmpeg process has expired, a new one will not be generated, so they only get whatever's left on the existing process.

There are several ways to fix that, but I haven't decided which one to use. But in the meantime, if the user simply reloads the page after the process expires, it will start a new stream.

On the plus side, however, because of the bandwidth savings, I was able to increase the maximum quality and the bitrate of the video.

I used a simple shell script on my office machine to check for an existing process and start one if needed:

#!/bin/bash
#
# Description: Checks for existing ffmpeg process and starts one if needed
#
ps -C ffmpeg
[ RETVAL -eq 0 ] && exit
[ RETVAL -ne 0 ] && nohup ffmpeg -f video4linux2 -i /dev/video0 -b:v 800K -r 30 -s 640x480 -t 600 http://[Remote IP]:8090/turtle.ffm
exit

On the Chicago side, I used a PHP SSH script to remotely execute the shell script on my local machine, and planted it in a hidden directory outside of /public_html/:

<?php
if (!function_exists("ssh2_connect")) die("Function ssh2_connect doesn't exist");
// Log in on port 22
if(!($con = ssh2_connect("[My Office IP]", 22))){
    echo "Unable to establish connection!\n";
} else {
    // Password login
    if(!ssh2_auth_password($con, "[user]", "[password]")) {
        echo "Authentication failed!\n";
    } else {
        // If successful login...
        echo "Connected!\n";

        // execute the script
        if (!($stream = ssh2_exec($con, "sh /home/[user directory]/ffmpeg-ck.sh" ))) {
            echo "Execution failed!\n";
        } else {
            // collect returning data from command
            stream_set_blocking($stream, true);
            $data = "";
            while ($buf = fread($stream,4096)) {
                $data .= $buf;
            }
            fclose($stream);
        }
    }
}
?>

And in the page itself, I entered a line to execute the SSH script:

exec('/usr/local/bin/php /home/[user directory]/[.hidden directory]/script-name.php &> /dev/null &');

So every time the page loads, it opens an SSH connection to the little server in my office, which in turn executes the shell script to start the video stream. It take between 1 and 3 seconds for this to happen; but they get higher picture quality in return.

One of the things I want to fix is the to restart the clock when a new user visits during an existing session. just killing and restarting the process would work, but that would be inelegant. So I'm going to ponder better ways.

Comments welcome.

Rich

 

[denverpilot]

Doesn't work on iPhone.

Also what's the big controversial new system manager thingy that replaced SysV good for on Cent7, if it can't do that restart shell script's job? I thought it was here to save the world from errant processes that needed bashed in the head. Hell, the thing apparently now has its own network stack. Soon it'll be full OS, maybe even complete with a "registry".

If you're going to run Fedora's garbage engineering that snuck into the Enterprise distro, might as well go whole hog. Hehehe.

(Just commenting because you said I could. Not because either of the above really bother me. And because I felt like putting another rant about the unnecessary death of SysV because some 20 year old doofus couldn't grok it, on the web somewhere new. Heh.)

 

[jesse]

Just change that SSH command to something like "touch /tmp/last-request-for-cam". Then run your ffmpeg-ck shell script from cron (or better yet write it to be a daemon that mostly sleeps and runs its logic every ten seconds). Change the logic of the bash script so that it checks the last modification date on that file and if it's greater than 10 minutes and ffmpeg is running kills the process. If it's less than 10 minutes and ffmpeg isn't running it will start ffmpeg.

Throw some javascript on your page so that it alerts every 9 minutes or so. If they close the alert then it sents an AJAX call to your website to bump the timer.

The above is pretty ideal because the page will never have to reload for the user and the user will never be interrupted provided they acknowledge the alert.

I'd probably toss SSH out the door completely and just write a few small web services. SSH is way overkill for what you're doing and is a pretty decent security risk.

 

[RJM62]

Doesn't work on iPhone.

Also what's the big controversial new system manager thingy that replaced SysV good for on Cent7, if it can't do that restart shell script's job? I thought it was here to save the world from errant processes that needed bashed in the head. Hell, the thing apparently now has its own network stack. Soon it'll be full OS, maybe even complete with a "registry".

If you're going to run Fedora's garbage engineering that snuck into the Enterprise distro, might as well go whole hog. Hehehe.

(Just commenting because you said I could. Not because either of the above really bother me. And because I felt like putting another rant about the unnecessary death of SysV because some 20 year old doofus couldn't grok it, on the web somewhere new. Heh.)

Thank you. I am pondering a different stream format and player for iThings and because I think Flash in general has seen its day. I also suspect that systemctl is part of the reason I had such fits trying to daemonize ffmpeg.

Rich

 

[RJM62]

Just change that SSH command to something like "touch /tmp/last-request-for-cam". Then run your ffmpeg-ck shell script from cron (or better yet write it to be a daemon that mostly sleeps and runs its logic every ten seconds). Change the logic of the bash script so that it checks the last modification date on that file and if it's greater than 10 minutes and ffmpeg is running kills the process. If it's less than 10 minutes and ffmpeg isn't running it will start ffmpeg.

Throw some javascript on your page so that it alerts every 9 minutes or so. If they close the alert then it sents an AJAX call to your website to bump the timer.

The above is pretty ideal because the page will never have to reload for the user and the user will never be interrupted provided they acknowledge the alert.

I'd probably toss SSH out the door completely and just write a few small web services. SSH is way overkill for what you're doing and is a pretty decent security risk.

Now that's an idea. I can daemonize my own script rather than the finicky ffmpeg, then install a lightweight http server like nginx or nanoweb on some obscure port on the video server and use curl on the receiving server to call it, but still use ffmpeg to push the stream. Thanks, Jesse.

Rich

 

[RJM62]

In the meantime, I have added a link to an AJAX call to restart the ffmpeg stream without refreshing the whole page.

Rich