Replies: 14 comments 13 replies
-
|
Generally, you process cues individually when they are found, it is not meant to return a collection of cues. cut and paste this into myfile.py #!/usr/bin/env python3
import json
import sys
from threefive import Stream
cue_list = []
def add_cue(cue):
cue_list.append(cue.get())
if __name__== '__main__':
strm = Stream(sys.argv[1])
strm.decode(func=add_cue)
print(json.dumps(cue_list))
then do python3 myfile.py video.tsand it will print it all together. |
Beta Was this translation helpful? Give feedback.
-
|
if you prefer using sed, this will add commas sed -e '2,$s/^{/,{/' test.json |
Beta Was this translation helpful? Give feedback.
-
|
@A-Snell |
Beta Was this translation helpful? Give feedback.
-
|
Sorry Adrian, got pulled into some other issues today. I will revisit this
by the end of the week!
Really appreciate your help though, I will definitely try these options and
see what works best.
Cheers
Alexander Snell
…On Wed, Jan 4, 2023 at 3:57 PM Adrian ***@***.***> wrote:
@A-Snell <https://github.com/A-Snell>
Talk to me Goose, did that fix it for you?
—
Reply to this email directly, view it on GitHub
<#74 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ATAEWRFPY22X277DGTHBOILWQXP33ANCNFSM6AAAAAATP5DEJY>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Closing the loop and asking one more thing... Firstly, here's the output I settled on: And the question: I have run MPTS' through this and am getting a lot of PIDs I don't care about. Is there a way to filter the parsing to only look at a specific PID or to only display a PID I care about natively? If not then I'll likely just run the JSON through JQ and do a search and remove that way. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Use the python shell, this is good python thing to know a@debian:~/build/clean/clean/clean/scte35-threefive$ pypy3
Python 3.9.16 (7.3.11+dfsg-2, Feb 06 2023, 16:52:03)
[PyPy 7.3.11 with GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>> from threefive import Stream
>>>> help(Stream) # this will show you all the methods and vars in the Stream class
>>>> help(Stream.decode_program) # You can also specify a particular method
Help on function decode_program in module threefive.stream:
decode_program(self, the_program, func=show_cue)
Stream.decode_program limits SCTE35 parsing
to a specific MPEGTS program.So you can specify a program to parse and ignore the others. >>>> from threefive import Stream
>>>> st = Stream("/home/a/mpegts/plp0.ts")
>>>> st.show()
Program: 1010 # A Program Number
Service: 01
Provider:
Pid: 1010
Pcr Pid: 1011
Streams: # PIDS in this program
Pid: 1011[0x3f3] Type: 0x1b AVC Video
Pid: 1012[0x3f4] Type: 0x3 Unknown
Pid: 1014[0x3f6] Type: 0x6 Unknown
Pid: 1015[0x3f7] Type: 0x86 SCTE35 Data
Program: 1030 # Another Program Number.
Service: 03
Provider:
Pid: 1030
Pcr Pid: 1031
Streams: # Pids in this Program
Pid: 1031[0x407] Type: 0x1b AVC Video
Pid: 1032[0x408] Type: 0x3 Unknown
Pid: 1034[0x40a] Type: 0x6 Unknown
Pid: 1035[0x40b] Type: 0x86 SCTE35 Datacall Strrean.decode_proxy wit h the program that contains the pid you want to parse >>>> from threefive import Stream
>>>> st =Stream("/home/a/mpegts/plp0.ts")
>>>> st.decode_program(the_program=1010) # Parse the Prgram 1010 that has SCTE-35 in Pid 1015 |
Beta Was this translation helpful? Give feedback.
-
|
try it like this #!/usr/bin/env python3
import json
import sys
from threefive import Stream
cue_list = []
def add_cue(cue):
cue_list.append(cue.get())
def get_args():
filename = None
if len(sys.argv) > 1:
filename = sys.argv[1]
the_program = 1 # 1 is the default Program
if len(sys.argv) > 2:
the_program = int(sys.argv[2]) # Second arg is the_program
return filename, the_program
def get_json(filename, the_program):
strm = Stream(filename)
print(f"strm File: {filename}") # f-strings are super cool
strm.decode_program(the_program=the_program, func=add_cue)
jsonForFile = json.dumps((cue_list), indent=4)
print(f"JSON: {jsonForFile}")
return jsonForFile
def write_json(filename, data):
err_mesg = "DID NOT FIND ANY SCTE DATA IN THE CAPTURE for Program {the_program}"
if len(data) < 10:
data = err_mesg
if "/" in filename:
filename = filename.rsplit("/", 1)[1]
outfile = f"{filename}-SCTE-35-DECODE.json" # f-strings are super cool
with open(outfile, "w") as myfile:
myfile.write(data)
# myfile.close() is not needed if you use "with"
if __name__ == "__main__":
filename, the_program = get_args()
data = get_json(filename, the_program)
write_json(filename, data) |
Beta Was this translation helpful? Give feedback.
-
|
run it like with the video and program as args.
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks very much Adrian. I was able to get that sample working, however for my purposes there are sometimes multiple SCTE 35 PIDs in a single program, so I have ended up parsing all and then using jq to go in afterwards and strip out all of the info_sections that match my selected PID. Which is a little bit inefficient but it works. Using your program selection method here is not going to be precise enough for my use case. Something else I noticed last week when running this was the packet_data/pts_ticks field was not accurate for the two capture files I was running it against. For the first of the files, the values did not make sense. Packet 1 and 3 are correct in the context of the stream, but the rest of the PTS values seem to be way off:
And for the 2nd capture, the time_signal messages were all the same value (4194303) and the splice_inserts 4095:
|
Beta Was this translation helpful? Give feedback.
-
|
Here are some sections from that first file:
|
Beta Was this translation helpful? Give feedback.
-
|
Hang on , let me try something for you. |
Beta Was this translation helpful? Give feedback.
-
|
try this: try this 35.py like
#!/usr/bin/env python3
import json
import sys
from threefive import Stream
cue_list = []
def add_cue(cue):
cue_list.append(cue.get())
def get_args():
filename = None
scte35_pids=[]
if len(sys.argv) > 1:
filename = sys.argv[1]
if len(sys.argv) > 2:
hexed = False
for i in sys.argv[2:]:
if i.startswith("0x") or i.startswith("0x"):
hexed =True
if hexed:
scte35_pids = [int(i,16) for i in sys.argv[2:]]
else:
scte35_pids =[int(str(i)) for i in sys.argv[2:]]
return filename,scte35_pids
def get_json(filename, scte35_pids):
strm = Stream(filename)
print(f"strm File: {filename}") # f-strings are super cool
print(f'SCTE-35 Pids: {scte35_pids}')
strm.decode_pids(scte35_pids=scte35_pids, func=add_cue)
jsonForFile = json.dumps((cue_list), indent=4)
print(f"JSON: {jsonForFile}")
return jsonForFile
def write_json(filename, data,scte35_pids):
err_mesg = f"DID NOT FIND ANY SCTE DATA IN THE CAPTURE for Pids {scte35_pids}"
if len(data) < 10:
data = err_mesg
if "/" in filename:
filename = filename.rsplit("/", 1)[1]
outfile = f"{filename}-SCTE-35-DECODE.json" # f-strings are super cool
with open(outfile, "w") as myfile:
myfile.write(data)
# myfile.close() is not needed if you use "with"
if __name__ == "__main__":
filename, scte35_pids = get_args()
data = get_json(filename, scte35_pids)
write_json(filename, data,scte35_pids) |
Beta Was this translation helpful? Give feedback.
-
|
Hey Adrian, this worked exactly as designed, thanks so much for deploying this so quickly. I am almost certain I will not be the only person that benefits from this. |
Beta Was this translation helpful? Give feedback.
-
|
How did it go? |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, I was travelling over the weekend, I will try to test this before the end of the week Adrian. Cheers! |
Beta Was this translation helpful? Give feedback.
-
|
Also curious if you have any thoughts on the pts_time being well off from reality that threeFive is writing for each info_section? |
Beta Was this translation helpful? Give feedback.
-
huh? |
Beta Was this translation helpful? Give feedback.
-
|
Are you talking about the in the pts from the packet data and pts from the splice command? "command": { "command_length": 5, "command_type": 6, "name": "Time Signal", "time_specified_flag": true, "pts_time": 67483.461489, "pts_time_ticks": 6073511534 },
"packet_data": { "pid": "0x1b", "program": 1, "pts_ticks": 6073052997, "pts": 67478.366633 } }Generally SCTE-35 packets are added 4 seconds before the actual splice time. Keep in mind, that PTS values are hard-coded in the SCTE-35, meaning that if you re-encode and change your PTS, , the SCTE-35 PTS does NOT update. In my opinion, that is a major flaw in the spec. |
Beta Was this translation helpful? Give feedback.
-
|
My question was that of the pts_ticks field in the packet_data that you have copied above:
My point was that for a couple of the info_sections in my decode, the value here seems to be correct, but the software is mis-reading the value for all other info sections. I.e. the value in this field is way off of the value in the actual SCTE 35 pts_time field, which makes the prerolls seem to be extremely long (like hours). All of this does raise a good question - how does this field get populated? Do you just find the nearest video packet and grab the PTS from there? Or is it computing a value using some other logic? Thanks again, Alex |
Beta Was this translation helpful? Give feedback.
-
|
The field is not populated, it's parsed from your SCTE-35 You may have the wrong values in it, but I assure you, threefive is not reading it incorrectly. As I explained in my last comment, Keep in mind, that PTS values are hard-coded in the SCTE-35, meaning that if you re-encode the video and change your PTS, , the SCTE-35 PTS does NOT update. In my opinion, that is a major flaw in the spec. |
Beta Was this translation helpful? Give feedback.
-
|
Are you talking about like this one ? { "info_section": { "table_id": "0xfc", "section_syntax_indicator": false, "private": false, "sap_type": "0x3", "sap_details": "No Sap Type", "section_length": 49, "protocol_version": 0, "encrypted_packet": false, "encryption_algorithm": 0, "pts_adjustment_ticks": 0, "pts_adjustment": 0, "cw_index": "0x0", "tier": "0x1", "splice_command_length": 20, "splice_command_type": 5, "descriptor_loop_length": 12, "crc": "0xcbdd9818" }, "command": { "command_length": 20, "command_type": 5, "name": "Splice Insert", "time_specified_flag": true, "pts_time_ticks": 6073511234, "break_auto_return": true, "break_duration": 600, "break_duration_ticks": 54000000, "splice_event_id": 3221225476, "splice_event_cancel_indicator": false, "out_of_network_indicator": true, "program_splice_flag": true, "duration_flag": true, "splice_immediate_flag": false, "unique_program_id": 0, "avail_num": 0, "avail_expected": 0 }, "descriptors": [ { "tag": 1, "descriptor_length": 10, "name": "DTMF Descriptor", "identifier": "CUEI", "preroll": 0, "dtmf_count": 4, "dtmf_chars": [ "4", "9", "4", "*" ] } ], "packet_data": { "pid": "0x1b", "program": 1, "pts_ticks": 4064427735,
|
Beta Was this translation helpful? Give feedback.
-
|
I was talking about the highlighted fields yes. You can see in my example how far off of each other those PTS' are in adjacent packets, which is not consistent with the PTS of the actual TS. My comment was that this is often inconsistent and I was wondering if you knew why. I will have to update threeFive on the server that hosts it (can take some effort, it is air-gapped in BAU) and let you know how I get on. Thanks for your assistance here Adrian. |
Beta Was this translation helpful? Give feedback.
-
|
Of course you have to update your system, I added code specifically for you. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, yes I have got it loaded locally where I can test (WSL/Ubuntu). I have updated my version: And run it against a captured TS, where threeFive-testing.py is a direct copy from your previous comment (#74 (comment)) The JSON that comes out of that command has PTS values that make sense in the Here is the list of the fields in question for the segmentation_messages in this break: Provider Advertisement End Distributor Advertisement Start Provider Placement Opportunity Start Provider Placement Opportunity End Distributor Advertisement End Provider Advertisement Start Provider Advertisement End Break End Chapter Start If I look at the And the last message is a chapter start for the start of the segment after the break. Look at the two PTS fields here: Provider Placement Opportunity Start "pts_time_ticks": 282310529 282310529 - 6212051315 = -5929740786 Chapter Start "pts_time_ticks": 289067279 289067279 - 6212017281 = -5922950002 If I compare that to the first message of the series, here's what we see: Provider Advertisement End pts_time_ticks: 282310529 282310529 - 281502279 = 808250 / 90000 = 8.9 seconds, which is how far in advance of the splice point this message appears in the TS. My question on this timing relationship then is how does the Hopefully that explains what I am seeing. Thanks very much! |
Beta Was this translation helpful? Give feedback.
-
|
Dude, you are wearing me out, SCTE-35 returns ticks for PTS, to make life easier, threefive includes the PTS as seconds and as ticks. I said verifty that it is a threefive issue before you bring it to me and you haven't shown me any proof that there is an error in threefive, I don't know what you expect me to do, I cant really tell you anything without seeing the stream, I'm not trying to be a dick, but I am spending a lot of my time helping you solve your problems, |
Beta Was this translation helpful? Give feedback.
-
|
I am just trying to understand how that |
Beta Was this translation helpful? Give feedback.
-
|
Latest Version is 2.3.91 (PTS fix for long AFC and Stream.decode_pids) |
Beta Was this translation helpful? Give feedback.
-
I am relatively new to Python programming so forgive my ignorance, but I am using the code from the test file '35.py':
`import sys
import threefive
if name == "main":
if len(sys.argv) > 1:
arg = sys.argv[1]
else:
arg = sys.stdin.buffer
strm = threefive.Stream(arg)
strm.decode()`
This works for me in that when passing a .ts file as a command line argument I get SCTE35 decoded and printed back to screen, however the output that comes back that I am writing into my textfile is not valid JSON. This is because there are no commas between successive objects. I have tried using sed and regex to wrangle some commas into those gaps, as well as trying to use the get_json function as part of threefive but cannot get this to work. This is an example of what I get back right now:
{ "info_section": { "table_id": "0xfc", "section_syntax_indicator": false, "private": false, "sap_type": "0x3", "sap_details": "No Sap Type", "section_length": 17, "protocol_version": 0, "encrypted_packet": false, "encryption_algorithm": 0, "pts_adjustment_ticks": 0, "pts_adjustment": 0.0, "cw_index": "0x0", "tier": "0xfff", "splice_command_length": 0, "splice_command_type": 7, "descriptor_loop_length": 0, "crc": "0x7f44f86a" }, "command": { "command_length": 0, "command_type": 7, "name": "Bandwidth Reservation" }, "descriptors": [], "packet_data": { "pid": "0x97e", "program": 140, "pts_ticks": 560784705, "pts": 6230.941167 } } { "info_section": { "table_id": "0xfc", "section_syntax_indicator": false, "private": false, "sap_type": "0x3", "sap_details": "No Sap Type", "section_length": 17, "protocol_version": 0, "encrypted_packet": false, "encryption_algorithm": 0, "pts_adjustment_ticks": 0, "pts_adjustment": 0.0, "cw_index": "0x0", "tier": "0xfff", "splice_command_length": 0, "splice_command_type": 7, "descriptor_loop_length": 0, "crc": "0x7f44f86a" }, "command": { "command_length": 0, "command_type": 7, "name": "Bandwidth Reservation" }, "descriptors": [], "packet_data": { "pid": "0x97e", "program": 140, "pts_ticks": 560784705, "pts": 6230.941167 } }What I would like is for there to be a comma in the space between the two objects:
{ "info_section": { "table_id": "0xfc", "section_syntax_indicator": false, "private": false, "sap_type": "0x3", "sap_details": "No Sap Type", "section_length": 17, "protocol_version": 0, "encrypted_packet": false, "encryption_algorithm": 0, "pts_adjustment_ticks": 0, "pts_adjustment": 0.0, "cw_index": "0x0", "tier": "0xfff", "splice_command_length": 0, "splice_command_type": 7, "descriptor_loop_length": 0, "crc": "0x7f44f86a" }, "command": { "command_length": 0, "command_type": 7, "name": "Bandwidth Reservation" }, "descriptors": [], "packet_data": { "pid": "0x97e", "program": 140, "pts_ticks": 560784705, "pts": 6230.941167 } },<-- comma here{
"info_section": { "table_id": "0xfc", "section_syntax_indicator": false, "private": false, "sap_type": "0x3", "sap_details": "No Sap Type", "section_length": 17, "protocol_version": 0, "encrypted_packet": false, "encryption_algorithm": 0, "pts_adjustment_ticks": 0, "pts_adjustment": 0.0, "cw_index": "0x0", "tier": "0xfff", "splice_command_length": 0, "splice_command_type": 7, "descriptor_loop_length": 0, "crc": "0x7f44f86a" }, "command": { "command_length": 0, "command_type": 7, "name": "Bandwidth Reservation" }, "descriptors": [], "packet_data": { "pid": "0x97e", "program": 140, "pts_ticks": 560784705, "pts": 6230.941167 } }Beta Was this translation helpful? Give feedback.
All reactions