TrackItem with ImageResource does not work in Timeline overlays

[ekurutepe]

Hello there,

I'm new to Cabbage and I'm struggling to figure out how to overlay image content on top of another video. Overlaying video on top another video works fine.

Here's what I'm trying to do:

let resource = ImageResource(image: CIImage(image: embed.image)!, duration: CMTime(seconds: 2.0, preferredTimeScale: 600))
let t = TrackItem(resource: resource)

t.startTime = embed.insertTime
t.duration = overlayDuration

t.videoConfiguration.frame = embed.frame
timeline.overlays = [t]

let compositionGenerator = CompositionGenerator(timeline: timeline)
guard let exporter = compositionGenerator.buildExportSession(presetName: AVAssetExportPreset1280x720) else {
       completion(nil)
       return
}

let outputFileURL = URL(fileURLWithPath: NSTemporaryDirectory() + UUID().uuidString + ".mp4")

exporter.outputURL = outputFileURL
exporter.outputFileType = .mp4
exporter.shouldOptimizeForNetworkUse = true

exporter.exportAsynchronously {
    switch exporter.status {
    case .completed:
        DispatchQueue.main.async {
            completion(exporter)
        }
    case .failed:
        DispatchQueue.main.async {
            completion(exporter)
        }
    case .cancelled:
        DispatchQueue.main.async {
            completion(exporter)
        }
    default:
        break
    }
}     

The video gets exported but does not contain the image overlay. I'd appreciate any help. Thanks!

[vitoziv]

For the question

If t.startTime = embed.insertTime is not zero, you should add a empty resource, and it's timeRange is 0 to t.startTime. Make sure the track is full.

like this |------------------|
not this |[empty]----------|

let resource = Resource()
resource.duration = t.startTime
resource.selectedTimeRange = CMTimeRange(start: CMTime.zero, end:  t.startTime)
let emptyTrackItem = TrackItem(resource: )
timeline.overlays = [emptyTrackItem , t]

Better way to add overlay image

timeline.overlays is for video overlay, it will add track to AVAsset

Another way, see the demo code

Cabbage/Cabbage/ViewController.swift

Lines 79 to 129 in 8fa17a1

	timeline.passingThroughVideoCompositionProvider = {
	let imageCompositionGroupProvider = ImageCompositionGroupProvider()
	let url = Bundle.main.url(forResource: "overlay", withExtension: "jpg")!
	let image = CIImage(contentsOf: url)!
	let resource = ImageResource(image: image, duration: CMTime.init(seconds: 3, preferredTimescale: 600))
	let imageCompositionProvider = ImageOverlayItem(resource: resource)
	imageCompositionProvider.startTime = CMTime(seconds: 1, preferredTimescale: 600)
	let frame = CGRect.init(x: 100, y: 500, width: 400, height: 400)
	imageCompositionProvider.videoConfiguration.contentMode = .custom
	imageCompositionProvider.videoConfiguration.frame = frame;
	imageCompositionProvider.videoConfiguration.transform = CGAffineTransform.init(rotationAngle: CGFloat.pi / 4)
	let keyframeConfiguration: KeyframeVideoConfiguration<OpacityKeyframeValue> = {
	let configuration = KeyframeVideoConfiguration<OpacityKeyframeValue>()
	let timeValues: [(Double, CGFloat)] = [(0.0, 0), (0.5, 1.0), (2.5, 1.0), (3.0, 0.0)]
	timeValues.forEach({ (time, value) in
	let opacityKeyframeValue = OpacityKeyframeValue()
	opacityKeyframeValue.opacity = value
	let keyframe = KeyframeVideoConfiguration.Keyframe(time: CMTime(seconds: time, preferredTimescale: 600), value: opacityKeyframeValue)
	configuration.insert(keyframe)
	})
	return configuration
	}()
	imageCompositionProvider.videoConfiguration.configurations.append(keyframeConfiguration)
	let transformKeyframeConfiguration: KeyframeVideoConfiguration<TransformKeyframeValue> = {
	let configuration = KeyframeVideoConfiguration<TransformKeyframeValue>()
	let timeValues: [(Double, (CGFloat, CGFloat, CGPoint))] =
	[(0.0, (1.0, 0, CGPoint.zero)),
	(1.0, (1.0, CGFloat.pi, CGPoint(x: 100, y: 80))),
	(2.0, (1.0, CGFloat.pi * 2, CGPoint(x: 300, y: 240))),
	(3.0, (1.0, 0, CGPoint.zero))]
	timeValues.forEach({ (time, value) in
	let opacityKeyframeValue = TransformKeyframeValue()
	opacityKeyframeValue.scale = value.0
	opacityKeyframeValue.rotation = value.1
	opacityKeyframeValue.translation = value.2
	let keyframe = KeyframeVideoConfiguration.Keyframe(time: CMTime(seconds: time, preferredTimescale: 600), value: opacityKeyframeValue)
	configuration.insert(keyframe)
	})
	return configuration
	}()
	imageCompositionProvider.videoConfiguration.configurations.append(transformKeyframeConfiguration)
	imageCompositionGroupProvider.imageCompositionProviders = [imageCompositionProvider]
	return imageCompositionGroupProvider
	}()

simplest and better way to add overlay image

let url = Bundle.main.url(forResource: "overlay", withExtension: "jpg")!
let image = CIImage(contentsOf: url)!
let resource = ImageResource(image: image, duration: CMTime.init(seconds: 3, preferredTimescale: 600))
let imageCompositionProvider = ImageOverlayItem(resource: resource)
timeline.passingThroughVideoCompositionProvider = imageCompositionProvider

[ekurutepe]

Thanks for the explanation.

The users of my app can embed multiple video and image overlays on a given video background. These embeds might be temporally and spatially overlapping with each other. My simple approach adding the TrackItems to the overlays array works great if the embeds are videos. That's why I was confused when it didn't work with images.

For now I'll try the empty track item approach but I'd be happy to send a PR to make this possible with TrackItems with ImageResources as well, if you think that's possible.