Add files using upload-large-folder tool

.gitattributes

@@ -36,3 +36,14 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 ffmpeg-6.1-amd64-static/qt-faststart filter=lfs diff=lfs merge=lfs -text
 ffmpeg-6.1-amd64-static/ffmpeg filter=lfs diff=lfs merge=lfs -text
 ffmpeg-6.1-amd64-static/ffprobe filter=lfs diff=lfs merge=lfs -text
+assets/demo/musk/musk.png filter=lfs diff=lfs merge=lfs -text
+assets/demo/monalisa/monalisa.png filter=lfs diff=lfs merge=lfs -text
+assets/demo/video1/video1.png filter=lfs diff=lfs merge=lfs -text
+assets/figs/musetalk_arc.jpg filter=lfs diff=lfs merge=lfs -text
+assets/demo/sun1/sun.png filter=lfs diff=lfs merge=lfs -text
+assets/demo/man/man.png filter=lfs diff=lfs merge=lfs -text
+data/video/sun.mp4 filter=lfs diff=lfs merge=lfs -text
+assets/demo/sun2/sun.png filter=lfs diff=lfs merge=lfs -text
+data/audio/yongen.wav filter=lfs diff=lfs merge=lfs -text
+data/video/yongen.mp4 filter=lfs diff=lfs merge=lfs -text
+data/audio/sun.wav filter=lfs diff=lfs merge=lfs -text

assets/BBOX_SHIFT.md

@@ -0,0 +1,26 @@
+## Why is there a "bbox_shift" parameter?
+When processing training data, we utilize the combination of face detection results (bbox) and facial landmarks to determine the region of the head segmentation box. Specifically, we use the upper bound of the bbox as the upper boundary of the segmentation box, the maximum y value of the facial landmarks coordinates as the lower boundary of the segmentation box, and the minimum and maximum x values of the landmarks coordinates as the left and right boundaries of the segmentation box. By processing the dataset in this way, we can ensure the integrity of the face.
+
+However, we have observed that the masked ratio on the face varies across different images due to the varying face shapes of subjects. Furthermore, we found that the upper-bound of the mask mainly lies close to the landmark28, landmark29 and landmark30 landmark points (as shown in Fig.1), which correspond to proportions of 15%, 63%, and 22% in the dataset, respectively.
+
+During the inference process, we discover that as the upper-bound of the mask gets closer to the mouth (near landmark30), the audio features contribute more to lip movements. Conversely, as the upper-bound of the mask moves away from the mouth (near landmark28), the audio features contribute more to generating details of facial appearance. Hence, we define this characteristic as a parameter that can adjust the contribution of audio features to generating lip movements, which users can modify according to their specific needs in practical scenarios.
+
+![landmark](figs/landmark_ref.png)
+
+Fig.1. Facial landmarks
+### Step 0.
+Running with the default configuration to obtain the adjustable value range.
+```
+python -m scripts.inference --inference_config configs/inference/test.yaml 
+```
+```
+********************************************bbox_shift parameter adjustment**********************************************************
+Total frame:「838」 Manually adjust range : [ -9~9 ] , the current value: 0
+*************************************************************************************************************************************
+```
+### Step 1.
+Re-run the script within the above range. 
+```
+python -m scripts.inference --inference_config configs/inference/test.yaml --bbox_shift xx  # where xx is in [-9, 9].
+```
+In our experimental observations, we found that positive values (moving towards the lower half) generally increase mouth openness, while negative values (moving towards the upper half) generally decrease mouth openness. However, it's important to note that this is not an absolute rule, and users may need to adjust the parameter according to their specific needs and the desired effect.

configs/inference/realtime.yaml

@@ -0,0 +1,10 @@
+avator_1:
+ preparation: False
+ bbox_shift: 5
+ video_path: "data/video/sun.mp4"
+ audio_clips:
+     audio_0: "data/audio/yongen.wav"
+     audio_1: "data/audio/sun.wav"
+ 
+  
+

configs/inference/test.yaml

@@ -0,0 +1,10 @@
+task_0:
+ video_path: "data/video/yongen.mp4"
+ audio_path: "data/audio/yongen.wav"
+
+task_1:
+ video_path: "data/video/sun.mp4"
+ audio_path: "data/audio/sun.wav"
+ bbox_shift: -7
+  
+

data/audio/sun.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3f163b0fe2f278504c15cab74cd37b879652749e2a8a69f7848ad32c847d8007
+size 1983572

data/audio/yongen.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2b775c363c968428d1d6df4456495e4c11f00e3204d3082e51caff415ec0e2ba
+size 1536078

data/video/sun.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9f240982090f4255a7589e3cd67b4219be7820f9eb9a7461fc915eb5f0c8e075
+size 2217973

data/video/yongen.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1effa976d410571cd185554779d6d43a6ba636e0e3401385db1d607daa46441f
+size 1870923

ffmpeg-6.1-amd64-static/model/vmaf_4k_rb_v0.6.2/vmaf_4k_rb_v0.6.2.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b367c22ef4d3369e6d609d81c96187cb6a7c23cf2b1203d580a072955487705b
+size 1058

ffmpeg-6.1-amd64-static/model/vmaf_4k_rb_v0.6.2/vmaf_4k_rb_v0.6.2.pkl.0002.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3ede915cf8718a16cd85264fdd2f3bd8e6ff258c1d400eca95465a0af3602cba
+size 21001

ffmpeg-6.1-amd64-static/model/vmaf_rb_v0.6.3/vmaf_rb_v0.6.3.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:250e31405a914bd2c493a77d5aaa6046d2841f5d2b458f61debc5b380b18dd5a
+size 1192

ffmpeg-6.1-amd64-static/model/vmaf_rb_v0.6.3/vmaf_rb_v0.6.3.pkl.0001.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3e7db1834c62e48e0da04e6c2671e7f19af77f3fd2afdd150cb59dd7083358aa
+size 17041

ffmpeg-6.1-amd64-static/model/vmaf_rb_v0.6.3/vmaf_rb_v0.6.3.pkl.0002.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:03021ce7009424a7ae9e654baa8bf2cfe0b2dfd51051c13261c953c5cbe46f4d
+size 17066

ffmpeg-6.1-amd64-static/model/vmaf_rb_v0.6.3/vmaf_rb_v0.6.3.pkl.0003.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fe82afada783967633870677ca367fb1e4564e46b25368254462abd1f277b1e4
+size 16968

ffmpeg-6.1-amd64-static/model/vmaf_rb_v0.6.3/vmaf_rb_v0.6.3.pkl.0006.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0839e94f1d9c66125a4ee9481d6b8d2bff40979bd766637ab8db5fdba64a61ed
+size 17260

ffmpeg-6.1-amd64-static/model/vmaf_rb_v0.6.3/vmaf_rb_v0.6.3.pkl.0007.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1f268b0f5eb5c1b3353c1821a1ad568c09ce07e40759a64a40db33363b8c96e5
+size 17009

ffmpeg-6.1-amd64-static/model/vmaf_rb_v0.6.3/vmaf_rb_v0.6.3.pkl.0008.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:18542324d8dc41a3d16db980afae11944cae845a735f9527f5f0492f9ab760f0
+size 16947

ffmpeg-6.1-amd64-static/model/vmaf_rb_v0.6.3/vmaf_rb_v0.6.3.pkl.0009.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3f090d7f6aa26b1a1065339e8441d24f4f7d310a201682b18369fe704e9386e6
+size 17209

ffmpeg-6.1-amd64-static/model/vmaf_rb_v0.6.3/vmaf_rb_v0.6.3.pkl.0010.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:def124ee54e7367c1d6bced55456f3e48df5dab624446b06a07b912a8d8fd742
+size 17128

ffmpeg-6.1-amd64-static/model/vmaf_rb_v0.6.3/vmaf_rb_v0.6.3.pkl.0011.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3268f5503e5a0e438642da8eb8c3cb0fca8848a1065e81381697448098c7e5f1
+size 16977

ffmpeg-6.1-amd64-static/model/vmaf_rb_v0.6.3/vmaf_rb_v0.6.3.pkl.0012.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a911555c138a9d05c46b1196de0279b6aaa4dc910bafe0a7ab6eb469fdd26bbf
+size 16910

ffmpeg-6.1-amd64-static/model/vmaf_rb_v0.6.3/vmaf_rb_v0.6.3.pkl.0013.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:691b42e87b4a7be454e2a2349b1894409c4c3c6d74d91f8d768c88e16a184f5a
+size 16835

ffmpeg-6.1-amd64-static/model/vmaf_rb_v0.6.3/vmaf_rb_v0.6.3.pkl.0014.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b878982625f4149a8b15fba8fd79b5a8968a9334ccc9dc1a2b83f998ab761e0f
+size 17169

ffmpeg-6.1-amd64-static/model/vmaf_rb_v0.6.3/vmaf_rb_v0.6.3.pkl.0015.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:19f39504d4b2f1309b4c76716a2cb701c4b1179f944ff272ce2b2c3c4c27e226
+size 17039

ffmpeg-6.1-amd64-static/model/vmaf_rb_v0.6.3/vmaf_rb_v0.6.3.pkl.0016.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9762de7f3ad0b44acd8ff7cc89a762c5b700cfd0059ae303be2ae13d1180df1c
+size 17276

ffmpeg-6.1-amd64-static/model/vmaf_rb_v0.6.3/vmaf_rb_v0.6.3.pkl.0017.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:df0b7fbce0a3a1f078b91d9a12295a1e33ff80d91ff3e0b4de2bdfc960370625
+size 17077

ffmpeg-6.1-amd64-static/model/vmaf_rb_v0.6.3/vmaf_rb_v0.6.3.pkl.0018.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:15d6d809353a65f9d57b7962ebd73a68dba650b490f8a38c6d48d7bd26b97dbb
+size 17371

ffmpeg-6.1-amd64-static/model/vmaf_rb_v0.6.3/vmaf_rb_v0.6.3.pkl.0019.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c54f2746647d4bd53acc49813ad6f519d7fafdd18820cbb81f8a1072cb840b1b
+size 16947

ffmpeg-6.1-amd64-static/model/vmaf_rb_v0.6.3/vmaf_rb_v0.6.3.pkl.0020.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cd5743c771d863fcb639512e64730e5b2c6ebc89e04766cbebe0c9201a0ce604
+size 17065

ffmpeg-6.1-amd64-static/model/vmaf_rb_v0.6.3/vmaf_rb_v0.6.3.pkl.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0c82d6c3881936b9babddb0ecd6b1f6b012a1008a3023f4ffe6ea42e7792eab1
+size 17004

models/dwpose/dw-ll_ucoco_384.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0d9408b13cd863c4e95a149dd31232f88f2a12aa6cf8964ed74d7d97748c7a07
+size 406878486

models/face-parse-bisent/79999_iter.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:468e13ca13a9b43cc0881a9f99083a430e9c0a38abd935431d1c28ee94b26567
+size 53289463

models/face-parse-bisent/resnet18-5c106cde.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5c106cde386e87d4033832f2996f5493238eda96ccf559d1d62760c4de0613f8
+size 46827520

models/musetalkV15/musetalk.json

@@ -0,0 +1,36 @@
+{
+  "_class_name": "UNet2DConditionModel",
+  "_diffusers_version": "0.6.0.dev0",
+  "act_fn": "silu",
+  "attention_head_dim": 8,
+  "block_out_channels": [
+    320,
+    640,
+    1280,
+    1280
+  ],
+  "center_input_sample": false,
+  "cross_attention_dim": 384,
+  "down_block_types": [
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D",
+    "DownBlock2D"
+  ],
+  "downsample_padding": 1,
+  "flip_sin_to_cos": true,
+  "freq_shift": 0,
+  "in_channels": 8,
+  "layers_per_block": 2,
+  "mid_block_scale_factor": 1,
+  "norm_eps": 1e-05,
+  "norm_num_groups": 32,
+  "out_channels": 4,
+  "sample_size": 64,
+  "up_block_types": [
+    "UpBlock2D",
+    "CrossAttnUpBlock2D",
+    "CrossAttnUpBlock2D",
+    "CrossAttnUpBlock2D"
+  ]
+}

models/sd-vae-ft-mse/diffusion_pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1b4889b6b1d4ce7ae320a02dedaeff1780ad77d415ea0d744b476155c6377ddc
+size 334707217

models/whisper/pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9607f98a2b22d9e229ae43c52ecea79dcede9e0c5cfae67e8da6eda86d8aac1d
+size 151095027

musetalk/models/unet.py

@@ -0,0 +1,51 @@
+import torch
+import torch.nn as nn
+import math
+import json
+
+from diffusers import UNet2DConditionModel
+import sys
+import time
+import numpy as np
+import os
+
+class PositionalEncoding(nn.Module):
+    def __init__(self, d_model=384, max_len=5000):
+        super(PositionalEncoding, self).__init__()
+        pe = torch.zeros(max_len, d_model)
+        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
+        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
+        pe[:, 0::2] = torch.sin(position * div_term)
+        pe[:, 1::2] = torch.cos(position * div_term)
+        pe = pe.unsqueeze(0)
+        self.register_buffer('pe', pe)
+
+    def forward(self, x):
+        b, seq_len, d_model = x.size()
+        pe = self.pe[:, :seq_len, :]
+        x = x + pe.to(x.device)
+        return x
+    
+class UNet():
+    def __init__(self, 
+                 unet_config,
+                 model_path,
+                 use_float16=False,
+                 device=None
+        ):
+        with open(unet_config, 'r') as f:
+            unet_config = json.load(f)
+        self.model = UNet2DConditionModel(**unet_config)
+        self.pe = PositionalEncoding(d_model=384)
+        if device != None:
+            self.device = device
+        else:
+            self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+        weights = torch.load(model_path) if torch.cuda.is_available() else torch.load(model_path, map_location=self.device)
+        self.model.load_state_dict(weights)
+        if use_float16:
+            self.model = self.model.half()
+        self.model.to(self.device)
+    
+if __name__ == "__main__":
+    unet = UNet()

musetalk/models/vae.py

@@ -0,0 +1,148 @@
+from diffusers import AutoencoderKL
+import torch
+import torchvision.transforms as transforms
+import torch.nn.functional as F
+import cv2
+import numpy as np
+from PIL import Image
+import os
+
+class VAE():
+    """
+    VAE (Variational Autoencoder) class for image processing.
+    """
+
+    def __init__(self, model_path="./models/sd-vae-ft-mse/", resized_img=256, use_float16=False):
+        """
+        Initialize the VAE instance.
+
+        :param model_path: Path to the trained model.
+        :param resized_img: The size to which images are resized.
+        :param use_float16: Whether to use float16 precision.
+        """
+        self.model_path = model_path
+        self.vae = AutoencoderKL.from_pretrained(self.model_path)
+
+        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+        self.vae.to(self.device)
+
+        if use_float16:
+            self.vae = self.vae.half()
+            self._use_float16 = True
+        else:
+            self._use_float16 = False
+
+        self.scaling_factor = self.vae.config.scaling_factor
+        self.transform = transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5])
+        self._resized_img = resized_img
+        self._mask_tensor = self.get_mask_tensor()
+        
+    def get_mask_tensor(self):
+        """
+        Creates a mask tensor for image processing.
+        :return: A mask tensor.
+        """
+        mask_tensor = torch.zeros((self._resized_img,self._resized_img))
+        mask_tensor[:self._resized_img//2,:] = 1
+        mask_tensor[mask_tensor< 0.5] = 0
+        mask_tensor[mask_tensor>= 0.5] = 1
+        return mask_tensor
+            
+    def preprocess_img(self,img_name,half_mask=False):
+        """
+        Preprocess an image for the VAE.
+
+        :param img_name: The image file path or a list of image file paths.
+        :param half_mask: Whether to apply a half mask to the image.
+        :return: A preprocessed image tensor.
+        """
+        window = []
+        if isinstance(img_name, str):
+            window_fnames = [img_name]
+            for fname in window_fnames:
+                img = cv2.imread(fname)
+                img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
+                img = cv2.resize(img, (self._resized_img, self._resized_img),
+                                     interpolation=cv2.INTER_LANCZOS4)
+                window.append(img)
+        else:
+            img = cv2.cvtColor(img_name, cv2.COLOR_BGR2RGB)
+            window.append(img)
+            
+        x = np.asarray(window) / 255.
+        x = np.transpose(x, (3, 0, 1, 2))
+        x = torch.squeeze(torch.FloatTensor(x))
+        if half_mask:
+            x = x * (self._mask_tensor>0.5)
+        x = self.transform(x)
+        
+        x = x.unsqueeze(0) # [1, 3, 256, 256] torch tensor
+        x = x.to(self.vae.device)
+
+        return x
+
+    def encode_latents(self,image):
+        """
+        Encode an image into latent variables.
+
+        :param image: The image tensor to encode.
+        :return: The encoded latent variables.
+        """
+        with torch.no_grad():
+            init_latent_dist = self.vae.encode(image.to(self.vae.dtype)).latent_dist
+        init_latents = self.scaling_factor * init_latent_dist.sample()
+        return init_latents
+    
+    def decode_latents(self, latents):
+        """
+        Decode latent variables back into an image.
+        :param latents: The latent variables to decode.
+        :return: A NumPy array representing the decoded image.
+        """
+        latents = (1/  self.scaling_factor) * latents
+        image = self.vae.decode(latents.to(self.vae.dtype)).sample
+        image = (image / 2 + 0.5).clamp(0, 1)
+        image = image.detach().cpu().permute(0, 2, 3, 1).float().numpy()
+        image = (image * 255).round().astype("uint8")
+        image = image[...,::-1] # RGB to BGR
+        return image
+    
+    def get_latents_for_unet(self,img):
+        """
+        Prepare latent variables for a U-Net model.
+        :param img: The image to process.
+        :return: A concatenated tensor of latents for U-Net input.
+        """
+        
+        ref_image = self.preprocess_img(img,half_mask=True) # [1, 3, 256, 256] RGB, torch tensor
+        masked_latents = self.encode_latents(ref_image) # [1, 4, 32, 32], torch tensor
+        ref_image = self.preprocess_img(img,half_mask=False) # [1, 3, 256, 256] RGB, torch tensor
+        ref_latents = self.encode_latents(ref_image) # [1, 4, 32, 32], torch tensor
+        latent_model_input = torch.cat([masked_latents, ref_latents], dim=1)
+        return latent_model_input
+
+if __name__ == "__main__":
+    vae_mode_path = "./models/sd-vae-ft-mse/"
+    vae = VAE(model_path = vae_mode_path,use_float16=False)
+    img_path = "./results/sun001_crop/00000.png"
+    
+    crop_imgs_path = "./results/sun001_crop/"
+    latents_out_path = "./results/latents/"
+    if not os.path.exists(latents_out_path):
+        os.mkdir(latents_out_path)
+
+    files = os.listdir(crop_imgs_path)
+    files.sort()
+    files = [file for file in files if file.split(".")[-1] == "png"]
+
+    for file in files:
+        index = file.split(".")[0]
+        img_path = crop_imgs_path + file
+        latents = vae.get_latents_for_unet(img_path)
+        print(img_path,"latents",latents.size())
+        #torch.save(latents,os.path.join(latents_out_path,index+".pt"))
+        #reload_tensor = torch.load('tensor.pt')
+        #print(reload_tensor.size())
+        
+
+    

musetalk/utils/__init__.py

@@ -0,0 +1,5 @@
+import sys
+from os.path import abspath, dirname
+current_dir = dirname(abspath(__file__))
+parent_dir = dirname(current_dir)
+sys.path.append(parent_dir+'/utils')

musetalk/utils/audio_processor.py

@@ -0,0 +1,99 @@
+import os
+import math
+import librosa
+import numpy as np
+import torch
+
+from einops import rearrange
+from transformers import AutoFeatureExtractor
+
+class AudioProcessor:
+    def __init__(self, feature_extractor_path="openai/whisper-tiny/"):
+        self.feature_extractor = AutoFeatureExtractor.from_pretrained(feature_extractor_path)
+        
+    def get_audio_feature(self, wav_path, start_index=0):
+        if not os.path.exists(wav_path):
+            return None
+        librosa_output, sampling_rate = librosa.load(wav_path, sr=16000)
+        assert sampling_rate == 16000
+        # Split audio into 30s segments
+        segment_length = 30 * sampling_rate
+        segments = [librosa_output[i:i + segment_length] for i in range(0, len(librosa_output), segment_length)]
+        
+        features = []
+        for segment in segments:
+            audio_feature = self.feature_extractor(
+                segment, 
+                return_tensors="pt",
+                sampling_rate=sampling_rate
+            ).input_features
+            features.append(audio_feature)
+
+        return features, len(librosa_output)
+    
+    def get_whisper_chunk(
+        self, 
+        whisper_input_features, 
+        device, 
+        weight_dtype, 
+        whisper, 
+        librosa_length,
+        fps=25,
+        audio_padding_length_left=2,
+        audio_padding_length_right=2,
+    ):
+        audio_feature_length_per_frame = 2 * (audio_padding_length_left + audio_padding_length_right + 1)
+        whisper_feature = []
+        # Process multiple 30s mel input features
+        for input_feature in whisper_input_features:
+            audio_feats = whisper.encoder(input_feature.to(device), output_hidden_states=True).hidden_states                
+            audio_feats = torch.stack(audio_feats, dim=2).to(weight_dtype)
+            whisper_feature.append(audio_feats)
+            
+        whisper_feature = torch.cat(whisper_feature, dim=1)
+        # Trim the last segment to remove padding
+        sr = 16000
+        audio_fps = 50
+        fps = int(fps)
+        whisper_idx_multiplier = audio_fps / fps
+        num_frames = math.floor((librosa_length / sr)) * fps
+        actual_length = math.floor((librosa_length / sr)) * audio_fps
+        whisper_feature = whisper_feature[:,:actual_length,...]
+        
+        # Calculate padding amount
+        padding_nums = math.floor(whisper_idx_multiplier)
+        # Add padding at start and end
+        whisper_feature = torch.cat([
+            torch.zeros_like(whisper_feature[:, :padding_nums * audio_padding_length_left]), 
+            whisper_feature, 
+            # Add extra padding to prevent out of bounds
+            torch.zeros_like(whisper_feature[:, :padding_nums * 3 * audio_padding_length_right])
+        ], 1)
+        
+        audio_prompts = []
+        for frame_index in range(num_frames):
+            try:
+                audio_index = math.floor(frame_index * whisper_idx_multiplier)
+                audio_clip = whisper_feature[:, audio_index: audio_index + audio_feature_length_per_frame]
+                assert audio_clip.shape[1] == audio_feature_length_per_frame
+                audio_prompts.append(audio_clip)
+            except Exception as e:
+                print(f"Error occurred: {e}")
+                print(f"whisper_feature.shape: {whisper_feature.shape}")
+                print(f"audio_clip.shape: {audio_clip.shape}")
+                print(f"num frames: {num_frames}, fps: {fps}, whisper_idx_multiplier: {whisper_idx_multiplier}")
+                print(f"frame_index: {frame_index}, audio_index: {audio_index}-{audio_index + audio_feature_length_per_frame}")
+                exit()
+                
+        audio_prompts = torch.cat(audio_prompts, dim=0)  # T, 10, 5, 384
+        audio_prompts = rearrange(audio_prompts, 'b c h w -> b (c h) w')
+        return audio_prompts
+
+if __name__ == "__main__":
+    audio_processor = AudioProcessor()
+    wav_path = "./2.wav"
+    audio_feature, librosa_feature_length = audio_processor.get_audio_feature(wav_path)
+    print("Audio Feature shape:", audio_feature.shape)
+    print("librosa_feature_length:", librosa_feature_length)
+    
+