Question / Help [solved] Connection to nginx+ffmpeg drops frequently

sneaky4oe · Dec 31, 2017

I have a linux transcoder server. It disconnects every few minutes with this message:

2017/12/31 02:53:36 [notice] 2400#0: signal process started
2017/12/31 02:54:17 [alert] 2404#0: unknown process 2405 exited on signal 9
2017/12/31 02:58:49 [alert] 2404#0: unknown process 2490 exited on signal 9
2017/12/31 02:59:08 [alert] 2404#0: unknown process 2577 exited on signal 9
2017/12/31 02:59:12 [alert] 2404#0: unknown process 2660 exited on signal 9
2017/12/31 02:59:16 [alert] 2404#0: unknown process 2661 exited on signal 9
2017/12/31 02:59:21 [alert] 2404#0: unknown process 2662 exited on signal 9
2017/12/31 02:59:43 [alert] 2404#0: unknown process 2663 exited on signal 9
2017/12/31 02:59:50 [alert] 2404#0: unknown process 2743 exited on signal 9
2017/12/31 02:59:54 [alert] 2404#0: unknown process 2744 exited on signal 9
2017/12/31 02:59:58 [alert] 2404#0: unknown process 2745 exited on signal 9
2017/12/31 03:00:53 [notice] 2757#0: signal process started
2017/12/31 03:01:01 [alert] 2761#0: unknown process 2762 exited on signal 9
2017/12/31 03:01:05 [alert] 2761#0: unknown process 2763 exited on signal 9
2017/12/31 03:01:59 [alert] 2761#0: unknown process 2765 exited on signal 9
2017/12/31 03:02:03 [alert] 2761#0: unknown process 2847 exited on signal 9
2017/12/31 03:03:22 [alert] 2761#0: unknown process 2848 exited on signal 9
2017/12/31 03:03:26 [alert] 2761#0: unknown process 2934 exited on signal 9
2017/12/31 03:03:32 [alert] 2761#0: unknown process 2935 exited on signal 9
2017/12/31 03:04:49 [notice] 3019#0: signal process started

OBS log:
https://gist.github.com/ed8b3ab379e2e03d35a2d837a7973a9e

The network is fine.

htop load on stream:

________________________

Solution that worked for me:
Install ubuntu 16.04 with HWE kernel (enabled upon installation).
Then use AMD videocard on 1080p/60 settings. More info in this post: https://obsproject.com/forum/thread...inx-ffmpeg-drops-frequently.78933/post-345766
FFMPEG can't go above faster preset for 1080/60 on a single ddr4 ram module and gives lots of artifacts. Tested on a ryzen build for one of my customers. Two modules may allow medium, overclocking may help too.

sam686 · Jan 1, 2018

A quick internet search, it might be caused by out of memory or other resource limits like process or thread limit.
https://www.google.com/search?q=linux+process+exited+9

Other ways this might happen, either manually or from another user/process, using: kill -9 (pid / process id number)

sneaky4oe · Jan 1, 2018

Cpu is loaded up to 50% only. Only 2 gigs/8 are used, so it's not memory blockage. I googled too and thought it might me a memory limit, but doesn't look like that...
Might be thread limit, but is 97 threads for ffmpeg too much for ryzen 1700? Especially if most of them are idling and only 16 are actively working.

There are no other users on this network and I didn't set it up for killing.

sam686 · Jan 1, 2018

Look at system logs for possible specific reason, like: dmesg
I am mostly looking at some comments shown in this website: https://stackoverflow.com/questions/726690/who-killed-my-process-and-why

pkv · Jan 1, 2018

did you set your workers to 1 ? post your nginx log.
The rtmp mopule does not work with multiple workers.

sneaky4oe · Jan 2, 2018

pkv said:
did you set your workers to 1 ? post your nginx log.
The rtmp mopule does not work with multiple workers.

Thanks, I'll get a look at it the next time I'll start getting drops.

pkv said:
did you set your workers to 1 ? post your nginx log.
The rtmp mopule does not work with multiple workers.

No, I didn't change worker number. It's set to 1 by default.

My logs for ubuntu nginx looks like this:

2018/01/01 07:07:36 [notice] 14457#0: signal process started
2018/01/01 07:28:23 [alert] 14461#0: unknown process 14498 exited on signal 9
2018/01/01 07:28:23 [alert] 14461#0: unknown process 14499 exited on signal 9
2018/01/01 17:41:13 [notice] 6377#0: signal process started
2018/01/01 17:54:47 [alert] 6389#0: unknown process 7152 exited on signal 9
2018/01/01 17:54:47 [alert] 6389#0: unknown process 7153 exited on signal 9
2018/01/01 17:56:07 [alert] 6389#0: unknown process 12617 exited on signal 9
2018/01/01 17:56:07 [alert] 6389#0: unknown process 12616 exited on signal 9
2018/01/01 21:05:29 [alert] 6389#0: unknown process 13337 exited on signal 9
2018/01/01 21:05:29 [alert] 6389#0: unknown process 13336 exited on signal 9
2018/01/01 21:12:52 [notice] 6693#0: signal process started
2018/01/01 21:18:05 [notice] 9295#0: signal process started
2018/01/01 22:24:50 [alert] 9299#0: unknown process 9330 exited on signal 9
2018/01/01 22:24:50 [alert] 9299#0: unknown process 9329 exited on signal 9

192.168.0.33 [01/Jan/2018:17:56:07 +0300] PUBLISH "receiver" "" "" - 61311584 601 "" "LNX.11,1,102,55" (1m 18s)
127.0.0.1 [01/Jan/2018:17:56:07 +0300] PUBLISH "transcoder" "1234" "" - 61311960 601 "" "LNX.11,1,102,55" (1m 18s)
127.0.0.1 [01/Jan/2018:17:56:07 +0300] PLAY "transcoder" "1234" "" - 930 61311774 "" "" (1m 18s)
127.0.0.1 [01/Jan/2018:17:56:07 +0300] PUBLISH "transcoded-1080" "1234" "" - 25101011 409 "" "" (1m 16s)
127.0.0.1 [01/Jan/2018:17:56:07 +0300] PLAY "transcoder" "1234" "" - 925 61311774 "" "" (1m 18s)
192.168.0.33 [01/Jan/2018:21:05:29 +0300] PUBLISH "receiver" "" "" - 3797130994 205753 "" "LNX.11,1,102,55" (3h 9m 19s)
127.0.0.1 [01/Jan/2018:21:05:29 +0300] PUBLISH "transcoder" "1234" "" - 3797600544 205513 "" "LNX.11,1,102,55" (3h 9m 19s)
127.0.0.1 [01/Jan/2018:21:05:29 +0300] PLAY "transcoder" "1234" "" - 649445 4066061337 "" "" (3h 9m 19s)
127.0.0.1 [01/Jan/2018:21:05:29 +0300] PUBLISH "transcoded-1080" "1234" "" - 762432891 409 "" "" (3h 9m 18s)
127.0.0.1 [01/Jan/2018:21:05:29 +0300] PLAY "transcoder" "1234" "" - 649095 4066061337 "" "" (3h 9m 19s)
192.168.0.33 [01/Jan/2018:22:24:50 +0300] PUBLISH "receiver" "" "" - 2515548760 72857 "" "LNX.11,1,102,55" (1h 6m 41s)
127.0.0.1 [01/Jan/2018:22:24:50 +0300] PUBLISH "transcoder" "1234" "" - 2515708399 72761 "" "LNX.11,1,102,55" (1h 6m 41s)
127.0.0.1 [01/Jan/2018:22:24:50 +0300] PLAY "transcoder" "1234" "" - 228350 1173535535 "" "" (1h 6m 40s)
127.0.0.1 [01/Jan/2018:22:24:50 +0300] PUBLISH "transcoded-1080" "1234" "" - 3168913711 409 "" "" (1h 6m 39s)
127.0.0.1 [01/Jan/2018:22:24:50 +0300] PLAY "transcoder" "1234" "" - 227685 1173535535 "" "" (1h 6m 40s)

Also this is my current config for workers:

rtmp {
server {
listen 1935;
chunk_size 4000; # max size 16777215

application receiver
{
live on;
meta copy;

# This sends stream to transcoder
push rtmp://localhost/transcoder/1234;

}

application transcoder
{
live on;
meta copy;

exec ffmpeg
-i rtmp://localhost/transcoder/1234

-b:v 6000k
-minrate 6000k
-maxrate 6000k
-bufsize 6500k

-vcodec libx264
-preset veryfast
-s 1920x1080
-sws_flags lanczos

-r 60 # FPS
-x264opts nal-hrd=cbr:force-cfr=1:keyint=120 # Double of your FPS

-acodec aac
-ab 160k
-strict -2

-f flv rtmp://localhost/transcoded-1080/1234;

# 540 periscope

exec ffmpeg
-i rtmp://localhost/transcoder/1234

-b:v 800k
-minrate 800k
-maxrate 800k
-bufsize 800k

-vcodec libx264
-preset faster
-s 960x540
-sws_flags lanczos

-r 30
-x264opts nal-hrd=cbr:force-cfr=1:keyint=60

-acodec aac
-ab 90k
-strict -2

-f flv rtmp://de.pscp.tv:80/x/************************;
}

application transcoded-1080
{
live on;
meta copy;

# Twitch transcoded
push rtmp://live-fra.twitch.tv/app/************************;

# youtube transcoded
push rtmp://b.rtmp.youtube.com:1935/live2/************************;

# Restream.io transcoded
push rtmp://eu-east.restream.io/live/************************;
}
}
}

sneaky4oe · Jan 4, 2018

I believe this is OBS problem itself as I can't stream stable all the time even to my localhost server...

pkv · Jan 4, 2018

try to downgrade to see if it makes a difference. There are several bug fixes waiting to be merged.
Tests also with other setups:
- x264 instead of NVENC;
- streaming without recording
- completely disable firewall
...
Post results with the logs

sneaky4oe · Jan 4, 2018

It didn't help with amd vce, nor with x264 or nvenc. So it's not a codec issue.
I've also seen a message that downscaling resolution to 1080 helped for obs, but I want to stream and record in 1440p and do transcoding on my own server having the record untouched.

Regarding downgrading, I tried downgrading to obs studio 17-something, and it didn't help either.

sneaky4oe · Jan 6, 2018

sam686 said:
Look at system logs for possible specific reason, like: dmesg
I am mostly looking at some comments shown in this website: https://stackoverflow.com/questions/726690/who-killed-my-process-and-why

Dmesg: https://gist.github.com/sneaky4oe/a8b12c83f328bf80a862785cfd190f3a

2018/01/06 22:59:25 [notice] 23503#0: signal process started
2018/01/07 00:58:21 [alert] 23530#0: unknown process 23823 exited on signal 9
2018/01/07 00:58:21 [alert] 23530#0: unknown process 23824 exited on signal 9
2018/01/07 00:58:26 [alert] 23530#0: unknown process 13516 exited on signal 9
2018/01/07 00:58:26 [alert] 23530#0: unknown process 13517 exited on signal 9

sneaky4oe · Jan 7, 2018

I believe this error code 9 may be related to ffmpeg, as I have 2 instances, and they crash at the same time in error log.

I launched manually this command:

ffmpeg -i rtmp://localhost/testerT/ -b:v 6000k -minrate 6000k -maxrate 6000k -bufsize 6500k -vcodec libx264 -preset faster -s 1920x1080 -sws_flags lanczos -r 60 -x264opts nal-hrd=cbr:force-cfr=1:keyint=120 -acodec aac -ab 160k -strict -2 -f flv rtmp://localhost/transcoded-1080/

It was working fine for about 10 minutes. At some point I got this error:

Code:

RTMP_ReadPacket, failed to read RTMP packet header

So it's either OBS don't sending it correctly, or Nginx failing in receiving.

pkv · Jan 7, 2018

can you post your FULL nginx config file ?
are you sure you've set :
worker_processes 1;
You told me you did but your errors suspiciously look like there are multiple workers active.
You posted only the rtmp server part of your nginx conf file ...

sneaky4oe · Jan 7, 2018

Ok, here it is. But as I said, I added only rtmp block and didn't touch the top part with html and stuff.

https://pastebin.com/RazUZnMY

pkv · Jan 7, 2018

did you try other outputs ? simple, advanced + advanced recording (custom ffmepg to url) ?
Also with upgraded obs ?
I have a setup similar to yours (obs 20.1.3 on win x64 to nginx with custom ffmpeg scripts or exec) and 0 issues.
Only difference is I encode with x264 instead of nvenc and I use either latest release or home compiled master or my own fork.

sneaky4oe · Jan 8, 2018

I did try outputs to localhost and back to my main streaming pc.
Latest OBS studio.
I also tried encoding with x264 - same result.
Do you stream at higher bitrates like I do? Are your gaming and streaming pcs intel-based?

pkv · Jan 8, 2018

i am on a 7700k; i stream at 720p or 1080p60 at most not 2K as you.

sneaky4oe · Jan 8, 2018

Well, I started streaming at 1080p lately. Didn't help.
Next I updated RTMP module on my nginx to this:
https://github.com/sergey-dryabzhinsky/nginx-rtmp-module

Didn't have enough time to test though...

pkv · Jan 8, 2018

i tested both arut vanilla and sergey; both work fine for me

sneaky4oe · Jan 9, 2018

With Sergey's I had no issues streaming for about 4 hours today. Could be more if my usb network card on the gaming pc to connect to the server directly didn't die midstream. >_<
Had to switch server back to router instead of PC. I'll need more tests to be done, but so far I can hope I've managed to get stable config (again).

sneaky4oe · Jan 11, 2018

TLDR: enabling gigabit connection, using sergey-dryabzhinsky rtmp module, adding some fixes to ffmpeg config.

So, I believe I've found the complex solution that now allows me to stream at 200 mbit/s flawlessly. At least 4 hours of csgo proved me right (normally it didn't last for more than 3 hours in the best days).
Ffmpeg starts twice (first time it's a birst of processes that shut down instantly without any error in log, then it actually starts working).

Setup:
Nginx+rtmp+ffmpeg modules:

Nginx rtmp module revision I am using right now: https://github.com/sergey-dryabzhin...tree/a01cc448ee2e8c747c6ff531a91958e044ac203d

Google drive: https://drive.google.com/open?id=1S4Q5N1LS74sX6TG407DfJ5b59eFtl3N6
My personal website (in case drive fails again): http://sneakyjoe.ru/deployment/nginx+rtmp_distros.zip

Add to /etc/rc.local (need to have ethtool installed):

exec 2> /tmp/rc.local.log # send stderr from rc.local to a log file
exec 1>&2 # send stdout to the same log file
set -x # tell sh to display commands before execution
sleep 5s
sudo ethtool -s enp35s0 speed 1000 ## This is for gigabit networks to enable higher speeds
/usr/local/nginx/sbin/nginx # Start Nginx

and make sure the gigabit connection is enabled with:

ethtool enp35s0

Transcoder's settings:

Code:

            application transcoder
                {
                    live on;
                    meta copy;

                # 1080 all sites
                    exec ffmpeg -i rtmp://localhost/transcoder/

                    -b:v 6000k
                    -minrate 5500k
                    -maxrate 6500k
                    -bufsize 7000k

                    -vcodec libx264
                    -preset faster
                    -s 1920x1080
                    -sws_flags bicubic
                  #  -vf unsharp=3:3:1.5 # Video sharpening

                    -r 60
                    -x264-params nal-hrd=cbr:force-cfr=1:keyint=120


                    -acodec aac
                    -ab 160k
                    -strict -2

                     -f flv rtmp://localhost/transcoded-1080/;

                # 560 periscope
                    exec ffmpeg -i rtmp://localhost/transcoder/

                    -b:v 740k
                    -minrate 600k
                    -maxrate 800k
                    -bufsize 800k

                    -vcodec libx264
                    -preset fast
                    -s 960x540
                    -sws_flags lanczos

                    -r 30
                    -x264-params nal-hrd=cbr:force-cfr=1:keyint=60

                    -acodec aac
                    -ab 90k
                    -strict -2

                    -f flv rtmp://localhost/transcoded-560/;
                }

I believe this works fine for now. Way better than everything I tried before.

Since my usb network card stopped working, the server was feeling fine connected to my gigabit router. Had to replace the cable again to make the connection actually go on gigabit speeds at last.

I'll update if there'll be any problems, since there's so little info about the same issue.

Question / Help [solved] Connection to nginx+ffmpeg drops frequently

sneaky4oe

Member

sam686

Member

sneaky4oe

Member

sam686

Member

pkv

sneaky4oe

Member

sneaky4oe

Member

pkv

sneaky4oe

Member

sneaky4oe

Member

sneaky4oe

Member

pkv

sneaky4oe

Member

pkv

sneaky4oe

Member

pkv

sneaky4oe

Member

pkv

sneaky4oe

Member

sneaky4oe

Member