Sign up Calendar Latest Topics
 
 
 


Reply
  Author   Comment   Page 1 of 2      1   2   Next
alekhka

Junior Member
Registered:
Posts: 15
Reply with quote  #1 
Hi,

I can't get sound of movie files play when using MovieGraphic adapter. I can play sounds perfectly when using AudioSound.

Is there a away to play sound of a video file in MovieGraphic?

Thanks!
0
Jaewon

Administrator
Registered:
Posts: 873
Reply with quote  #2 
Movie objects in ML do not process audio. Use with sound objects to play audio.
0
alekhka

Junior Member
Registered:
Posts: 15
Reply with quote  #3 
Thank you so much!

We extracted the audio from the movie and played it with AudioSound adapter.

But we are seeing a lot of sync issue between the movie and the sound. We see that there are lot of skipped frames in the video (60-70 per 2 minute video which probably is causing the de-sync, we are not sure).

I have attached the conditions file, timing script and the video. We would be very grateful if you could look at it.

Please let us know if you find a solution for this, thank you so much!
0
Jaewon

Administrator
Registered:
Posts: 873
Reply with quote  #4 
For most computers, decoding a full HD source without frame loss is almost impossible, even when the computer is not doing anything else. Change the screen resolution to a smaller size and resize the movie.

-----

And the conditions file should be tab-delimited. You used spaces between task objects, which caused an error in reading the file. Try the attached.

 
Attached Files
txt Movie_Conditions.txt (142 Bytes, 3 views)

0
alekhka

Junior Member
Registered:
Posts: 15
Reply with quote  #5 
Thank you so much!
Do you think we can improve the performace if we use a better GPU?

We tried a small resolution movie. Now there is almost no skipped frames yet the audio lag is present. The lag becomes more pronounced when loading large files (2hr long movie in very small resolution and low bitrate).

Any recommendations?

0
Jaewon

Administrator
Registered:
Posts: 873
Reply with quote  #6 
I actually found something and am working on it. The issue occurs only at the first trial, because some improvement I previously made affects the initialization of trials. Now I need to straighten up the code so that this kind of thing can be easily detected. I will upload a new package soon.

By the way, ML discards skipped frame, by default, so that it keeps up with audio timing. In a natural video like yours, a few skipped frames do not affect the quality of video much. So, if you want, you can go back to the full HD video, after I fix this issue.
0
alekhka

Junior Member
Registered:
Posts: 15
Reply with quote  #7 
Thank you again!

We actually found out by accident that it affects only the first trial.

Strangely, this works: (Note how the video is not even loaded in the first trial)
if first_trial
  do_nothing
else
  load_and_play_video
 

For small videos (we tested a 2min one) audio sync is fine in the second trial. But for a large file (2hrs), the sync issue is present even in second trial, it also takes a lot of time to start playing the video.

Do you have any recommendations for large length files?
0
Jaewon

Administrator
Registered:
Posts: 873
Reply with quote  #8 
Please try the new package I just uploaded and let me know if the problems you found persist.

I haven't tried 2-hour long movies. Can't you send me your movie?
0
alekhka

Junior Member
Registered:
Posts: 15
Reply with quote  #9 
Will try and let you know at the earliest. Thank you so much again!

Will try to upload the file (it is pretty big in size). Although, I think any full length movie should cause the problem.
0
Jaewon

Administrator
Registered:
Posts: 873
Reply with quote  #10 
How big is the file? You may upload it to a cloud service and share with me.
0
alekhka

Junior Member
Registered:
Posts: 15
Reply with quote  #11 
Okay, will send it to you tomorrow.

Thanks again!
0
kpan

Junior Member
Registered:
Posts: 5
Reply with quote  #12 
Hi Jaewon, 

We just shared a zipped folder with the files on Google Drive with your NIH email - it's called "MovieTesting_LongMovie". Thanks so much for your help, and let us know if you have any problems accessing it!
0
Jaewon

Administrator
Registered:
Posts: 873
Reply with quote  #13 
I don't have any particular issue in playing the movie. However, my computer doesn't have enough memory to load the wav file (1.72 GB). Maybe I should stream the audio as well, but I don't think I can work on it anytime soon.
0
Jaewon

Administrator
Registered:
Posts: 873
Reply with quote  #14 
Hi alekhka and kpan,

Please try the new package that I just uploaded. I played the movie for about 10 minutes and didn't see any noticeable async. Hopefully you guys can do more through tests.

http://forums.monkeylogic.org/post/nimh-monkeylogic-2-8444337?pid=1307863341
0
kpan

Junior Member
Registered:
Posts: 5
Reply with quote  #15 
Thank you Jaewon! We tried this and it worked really well for ~1.5 hours.
0
alekhka

Junior Member
Registered:
Posts: 15
Reply with quote  #16 
Hi Jaewon!

It works well, it is lot more fast and stable now, thank you so much!

I was wondering if there was a callback after each frame was rendered? 

I want to know exactly when each frame was rendered and send a digital code/save the timestamp. Is it possible with MovieGraphic adapter?


0
Jaewon

Administrator
Registered:
Posts: 873
Reply with quote  #17 
See the FrameMarker.
ftp://helix.nih.gov/lsn/monkeylogic/ML2_doc/runtimefunctions.html#FrameMarker
0
alekhka

Junior Member
Registered:
Posts: 15
Reply with quote  #18 
Thank you very much!

This is tied to the screen refresh and not movie frames, as in, the code is sent every time screen is refreshed even if there is a skipped frame. We don't see a straightforward way to see which movie frame was rendered at that instant. 

Is there anything in MovieGraphic which keeps track of the frames of the movie?

Also, is there a way to pass the movie to MovieGraphic from the conditions file. MovieGraphic now works only when filename is passed. 

Thanks again!
0
Jaewon

Administrator
Registered:
Posts: 873
Reply with quote  #19 
It is tied to movie frames, although the code may not be sent if the target frame is skipped.

I am not sure what you want to do here. What is the code you want to send? Do you want to send it every frame?

You need to provide the MGL object ID, not the TaskObject number.

mov.MovieList = {TaskObject(2).ID, [0 0]};  % TaskObject #2

0
alekhka

Junior Member
Registered:
Posts: 15
Reply with quote  #20 
Yeah, I want to send a code for every frame (right after every frame of the movie is presented). We actually want to go from 1 to 100 and then start from 1 again, that is, send code from 1 to 100 and then repeat. For now, we placed a marker for the 10th and the 200th frame like this -
fm.FrameEvent = [10 2; 200 4];  

We see that the first code (for 10th frame) is at ~750ms and the second (for 200th) at ~3.9s. This means that the frames are presented at 60fps while the video is 29.97fps. This led us to believe that these timestamps are tied to screen refresh rate and not the video frame rate.
Interestingly, for a 10s 29.97fps video, get_movie_duration( ) returns 10s and ~600 frames. I have verified with FFmpeg that the video indeed has only 300 frames.

Is there something we are missing here?

Thank you so much again for taking the time to reply to our queries, we deeply appreciate it!
0
Jaewon

Administrator
Registered:
Posts: 873
Reply with quote  #21 
Unfortunately it doesn't always work to calculate the current movie position by multiplying the video frame rate and the number of frames. It is not a huge violation that a movie file has less than 30 frames per sec although its header says it is a 30-fps movie. Each video frame in the movie file has a timestamp that determines when the frame should be presented. That timestamp is more reliable than the video frame rate. The video frame rate is just a redundant field in the header and it is up to individual applications whether to respect that value or not.

p.scene_frame() in the FrameMarker adapter (or any adapter) returns the time elapsed from the scene start in terms of the number of screen refreshes and is also used internally to calculate the current movie position. So I think you can stamp the return value of p.scene_frame() every screen refresh and use it to calculate the video frame number later.

If you present a 10-s movie on a 60-Hz monitor, there will be 600 screen frames and their time will be:

(0:599) * (1/60)  % Count 600 frames from 0 so that the time of the very first frame becomes 0.

Let's say the video is 30fps. Then the video frame number to be shown each screen refresh is:

floor((0:599)*(1/60)/(1/30))+1  % This produces video frame numbers that start from 1.

The above formula works only when the movie contains all 300 video frames with no missing one. Now you can put the stamped value of p.scene_frame() in the formula, instead of (0:599), and get the video frame number presented.

0
alekhka

Junior Member
Registered:
Posts: 15
Reply with quote  #22 
Thank you so much Jaewon!

We are trying to send a behavioral code every screen refresh by modifying the MovieGraphic adapter, somewhat like this:

function continue_ = analyze(obj,p)

            continue_ = analyze@mlstimulus(obj,p);
            obj.Success = obj.Adapter.Success;
            frame_number = floor(p.scene_frame()*30/60)+1;
            fprintf('%d : %.6f\n', frame_number, p.scene_time());
            p.EventMarker = 21;

end

1. But we see that the ITI has gotten too high, which we think is because it saves the data at the end of each trial. But we don't see such a overhead when a joystick is connected (recording at 1000Hz). Is the way we have done the recommended way to do it?

2. We see that the scene_time printed out and the code_time of the code are vastly different. We understand that the code is sent with the next screen refresh, so shouldn't they be off by only a little?

3. We also notice that skipped frames happen more frequently (sometimes skipping 200-300 at once) now that we are sending so many behavioral codes. Is there a more efficient way to send the codes?

4. We find the ability to send a code on skipped frames really helpful. But it seems like when many frames are skipped at once, the code is only sent once. Is there a way to send multiple codes so that we know how many frames were skipped?

Thank you so much for all your help!


0
Jaewon

Administrator
Registered:
Posts: 873
Reply with quote  #23 
I believe the timing problems described in your question 2 & 3 are because you inserted "fprintf". Printing something on the MATLAB command window takes a long time. You can't measure anything correctly, having it in your code.

Regarding to the question 4, are you talking about the code 13? If so, I would like to mention that you can retrieve the number of skipped frames from your data file. The skipped frame info is saved to the UserVars.SkippedFrameTimeInfo field, whether you activate the code 13 or not, and it contains the following information.

    SkippedFrameTimeInfo(:,1) - trialtime that the frame was supposed to be presented at
    SkippedFrameTimeInfo(:,2) - trialtime that the frame was actually presented at
    SkippedFrameTimeInfo(:,3) - The number of frames skipped
    SkippedFrameTimeInfo(:,4) - Frame length (msec)
    SkippedFrameTimeInfo(:,5:end) - Additional time information

I don't think sending multiple codes for skipped frames is a good idea. Getting skipped frames means that you are already behind the schedule and you don't want to be stuck there until all codes are sent.

If you activated the code 13 by setting TrialRecord.MarkSkippedFrames true and had many skipped frames, that may explain the long ITI. MATLAB GUI is slow, so updating the Timeline panel may take a long time, when there are multiple events to draw. You can choose not to update the Timeline panel by setting TrialRecord.DrawTimeLine to false.
0
Jaewon

Administrator
Registered:
Posts: 873
Reply with quote  #24 
By the way, I thought you wanted to do something like:

p.EventMarker = 1000 + mod(p.scene_frame(),100);

Then, you can do both calculating the frame number and counting the number of skipped frames.
0
Jaewon

Administrator
Registered:
Posts: 873
Reply with quote  #25 
Somehow a line was missing in both ImageGraphic and MovieGraphic. Please update.

----- draw() of ImageGraphic & MovieGraphic -----
...
function draw(obj,p)
    draw@mlstimulus(obj,p);
    if ~obj.Triggered && obj.Success
    ...
    end

end
...
----- End of code -----
0
Previous Topic | Next Topic
Print
Reply

Quick Navigation:

Easily create a Forum Website with Website Toolbox.