Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

近期计划是什么?| What are the short-term plans? #280

Open
hengtuibabai opened this issue May 9, 2023 · 39 comments
Open

近期计划是什么?| What are the short-term plans? #280

hengtuibabai opened this issue May 9, 2023 · 39 comments
Labels
discussion This will not be worked on

Comments

@hengtuibabai
Copy link

停更了吗?近期计划是什么?
感觉离好用还差一点点了啊,停更的话太可惜了!

Has it stopped? What are the short-term plans?
I feel like just a little short of being useful. It would be a pity if I stopped the watch!

@vinthony
Copy link
Collaborator

vinthony commented May 10, 2023

hi, 最近再肝paper,可能会在月底或者6月份开始继续更新。目前已经完成的部分和正在做的部分,比如:

@vinthony vinthony added the discussion This will not be worked on label May 10, 2023
@vinthony
Copy link
Collaborator

vinthony commented May 10, 2023

any new suggestions are also welcome : )

@vinthony vinthony pinned this issue May 11, 2023
@vinthony vinthony changed the title 停更了吗?近期计划是什么?| Has it stopped? What are the short-term plans? 近期计划是什么?| What are the short-term plans? May 11, 2023
@2793145003
Copy link

之前看到 有人用 Diffusion model,用 Diffusion model 的话也许能做到原图分辨率?(不懂瞎说)

@vinthony
Copy link
Collaborator

diffusion-based 估计会很慢,不过face vid2vid也很慢就是了。质量上我从paper里没看到很大的差距因为他的diffusion不是SD,是自己训的。这里有一个备选是 先用sadtalker生成keypoints,再用pre-trained 的controlnet-face去解。

@lliang2003
Copy link

OpenTalker WEBUI, 集成Sadtalker和video-retalking的技术,做到输入图像和视频进行驱动和编辑。

@canghaiyunfan
Copy link

请问如何实现只针对头进行动作?

@Niutonian
Copy link

I love your project,
I think a good way to control ref_pose and ref_eyebllink would be awesome,
Another thing that I would love to see implemented is "Idle animation", in case no audio is played it goes through a preset sequence of blink and move without looking too weird

@MoroseYu
Copy link

MoroseYu commented Jun 4, 2023

不好停更啊,很期待~

@vinthony
Copy link
Collaborator

vinthony commented Jun 6, 2023

cool! thanks for your advise! will woking on it.

@2793145003
Copy link

之后会解决脸色苍白双眼无神的问题吗?(还是我用的方法不对?
quick_demo里的full3,size=512,出来的结果脸会变得很白(去掉gfpgan也很白)
眼睛也一直虚着,去掉still之后可以眨眼但又没法贴回原图

@vinthony
Copy link
Collaborator

I love your project, I think a good way to control ref_pose and ref_eyebllink would be awesome, Another thing that I would love to see implemented is "Idle animation", in case no audio is played it goes through a preset sequence of blink and move without looking too weird

We have update this feature in #386, however, more work need to be done to make it better.

@vinthony
Copy link
Collaborator

vinthony commented Jun 12, 2023

这个我也发现了,可能是和训练数据有关。我找时间再训练一下模型。

之后会解决脸色苍白双眼无神的问题吗?(还是我用的方法不对? quick_demo里的full3,size=512,出来的结果脸会变得很白(去掉gfpgan也很白) 眼睛也一直虚着,去掉still之后可以眨眼但又没法贴回原图

@zyl280505776
Copy link

挺好的,期待持续迭代

@jyzd111
Copy link

jyzd111 commented Jun 17, 2023

大佬厉害,我用的国外的网站生成的和你效果差不多,你的还能自己调整参数。期待早日上2D,

@grazder
Copy link

grazder commented Jun 23, 2023

what are your plans on cartoon images? like in makeittalk

@vinthony
Copy link
Collaborator

A lightweight facerender is added for generation, which might be working in real-time on GPU and 100x faster on Macbook. See the discussion #457.

@vinthony
Copy link
Collaborator

what are your plans on cartoon images? like in makeittalk

will try to add something like: https://github.com/pkhungurn/talking-head-anime-3-demo

@Kedreamix
Copy link

你好,我想问一下,有计划开源训练的代码么,也就是每一part的训练代码

@FranM2030
Copy link

I've noticed that you're able to provide a head pose reference video, could we do the same for a half body. Provide a reference video for half body that drives the upper body movement along with the head? Just he head is a bit limited.

@xyyyuuan
Copy link

xyyyuuan commented Aug 9, 2023

amd的显卡是不是只能跑在cpu上,amd不动

@ifredom
Copy link

ifredom commented Aug 23, 2023

感谢大兄弟,就差一点就很好用了,有空更更,不要鸽啦

@skyliwq
Copy link

skyliwq commented Aug 30, 2023

加油加油

@warycat
Copy link

warycat commented Oct 24, 2023

加油加油,这个东西太有用了,我要把它集成到我的app里。

@Tybost
Copy link

Tybost commented Nov 11, 2023

Still crossing my fingers for anime head support. ;)

@slavakurilyak
Copy link

When can we expect the next release? The last release was made 168 days ago.

@creepcat-gh
Copy link

大佬厉害,确实非常好用

@zjy-2020
Copy link

大佬,网盘下载的压缩包李没有.pth文件和BFM, hub文件了吗
就是checkpoints/auido2exp_00300-model.pth,checkpoints/auido2pose_00140-model.pth, checkpoints/epoch_20.pth等文件。
没有话程序老是报错

@rucieryi369
Copy link

会有微调的代码吗?谢谢

@denvey
Copy link

denvey commented Dec 8, 2023

OpenTalker WEBUI

请问这个在哪里呀

@XayerMorgan
Copy link

Any plans for continued development. I was using this for a while, but suddenly it stopped working. Can't find the root cause, because automatic1111 and foocus work great. Standalone or automatic1111 give same errors. Would love to see this continue, and it's a remarkable piece of work.

@oisilener1982
Copy link

Sadly Devs are very busy in other projects :(

@ck1123456
Copy link

您好,请问我提供中文音频,生成的视频不是自己提供的声音,求指教

1 similar comment
@ck1123456
Copy link

您好,请问我提供中文音频,生成的视频不是自己提供的声音,求指教

@oisilener1982
Copy link

When can we expect the next release? The last release was made 168 days ago.

Most likely there will no further updates. The last update was 9 months ago. With the coming release of EMO and VASA-1 sadtalker might really be dead, however they are not released yet

The other alternative is v-express by tencent but sadtalker is way better,

@NinoNeumann
Copy link

Now there might be something even better—hallo (https://github.com/fudan-generative-vision/hallo), and the exciting part is that this project is open-source.

@oisilener1982
Copy link

Hallo is taking so long to render :( We need to upgrade to Rtx 5090 once it is released

@gg22mm
Copy link

gg22mm commented Oct 12, 2024

看好这个项目,期待升级更牛的版本出来。 ~~ 虽然目前在图片生成视频已是 number on

@gg22mm
Copy link

gg22mm commented Oct 12, 2024

Now there might be something even better—hallo (https://github.com/fudan-generative-vision/hallo), and the exciting part is that this project is open-source.

图片
这说的这个不支持中文搞个毛线

@gg22mm
Copy link

gg22mm commented Oct 12, 2024

hi, 最近再肝paper,可能会在月底或者6月份开始继续更新。目前已经完成的部分和正在做的部分,比如:

* [x]  支持size参数可以控制输出图的分辨率,训了一个512x512的模型face render还在测试之中,期望可以干掉人脸增强器。

* [x]  mannually的控制crop的区域,可以得到更加自定义的结果,WEBUI。

* [x]  WEBUI更多功能,比如refpose等。

* [x]  接受了两种不同的自动crop方式,能够复现v0.0.1版本的效果。

* [x]  减少了一些依赖文件,比如dlib,用现在的方法更便于安装。

* [x]  加速模型和优化checkpoints,可能会不需要下载那么大的模型。

* [ ]  更加解耦的mappingnet的训练,可能可以支持只针对头进行动作。

* [x]  A simpler facerender model (or the TensorRT support) will be included to support faster generation: [A light-weight FaceRender is support for more than 100x faster rendering on Mac OS. #457](https://github.com/OpenTalker/SadTalker/discussions/457)

* [ ]  text-geneartion-webui

* [ ]  anime-generator.

* [ ]  OpenTalker WEBUI, 集成Sadtalker和video-retalking的技术,做到输入图像和视频进行驱动和编辑。

* [ ]  Fix API problem, [是否支持API调用的方式生成视频? #379](https://github.com/OpenTalker/SadTalker/issues/379),[[bug] [gradio] Web API not working #374](https://github.com/OpenTalker/SadTalker/issues/374) , [这是个特别牛的项目,请问SD那边能用api调用吗? #251](https://github.com/OpenTalker/SadTalker/issues/251) ,

* [ ]  FPS, [Frame Per Second Option request. #294](https://github.com/OpenTalker/SadTalker/issues/294),

我感觉以后升级的话(可以往这方面发展,这样就不会有头部与身体衔接问题):
1、直接生成全省图,现在的逻辑是先生成半身图如:
1728719472301

2、再经过full.才放大(就会有衔接问题,头部与身体会些错位,有抖动错位)

4.3_full.mp4

3、最后经过enhanced提升效果

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
discussion This will not be worked on
Projects
None yet
Development

No branches or pull requests