Is there a way to download a version without any of the generative ai features? The idea of having them even if i won't use them feels icky to me, but this is the most recommended face tracker for inochi models.
I don't think so but you can try Inochi Session though you'll need to download face tracking softwares/apps. Face tracking softwares/apps usually has some form of AI but luckily some dosen't- not that I could find any with a quick google search but some outdated ones hopefully exsist, else, code your own. Most run using Google Mediapipe which has AI(unfortunately, it used to not have it but now it does) or others which usually has AI too. Face-tracking without AI at all runs on older, more traditional algorithms and hardwares which also require in turn alot more expenses even if you do find one without AI so I'd rather it include some form of AI. Traditional face tracking without AI require : Logitech C920s HD Pro Webcam(225SGD cheapest), 2x Micro servo motors(for pan-tilt, about 14SGD cheapest), Pololu 1350 USB Servo Controller(around 50SGD cheapest), Pan-Tilt Bracket Kit(36SGD cheapest), Ring lights, 3x soft lights, background light or greenscreen and of course the technical knowledge to put all that together including the pan-tilt system since most are hardware parts. Also the lights have to be neutral white lights, no color or gradients. Face tracking without AI requires specific specific hardware like those because it relies on tracking your face movements using geometric algorithms, etc which relies on high-contrast lighting in a dark room so the computer can calculate your face positioning and most hardware have AI chips nowadays. I'm by no means an expert so the information I listed may be lacking or incorrect and most of which are just from a quick google search but I do have a certificate for coding AI applications which was mandatory ALP(applied learning program) in my secondary school which was 5 or 6 years ago, closer to 7 years now(yes, AI has been around since way before the whole AI, Anti-AI thing.) As an animator, I do not support AI whatsoever but it's unavoidable. Vtube studio uses AI for face tracking as well so the only AI-free solution is to use Vbridger with a Live2D model which is extremely costly with Live2d models costing upwards 4k. Most hardware nowadays have some form of an AI chip though linux or linux mint is an AI free solution though certain games don't run on linux.
My problem is more the generative forms of at (Think LLM/Chatbots and image generation) and while i know i don't HAVE to use the provided chatbot, it just doesn't sit very comfortably with me just having it on my devices without my say. I appreciate the alternatives tho, it does help a ton and i probably will end up using an alternative.
It's still being worked on, however you should temper your expectation especially if you're not using it with models made by Mannequin Character Generator.
This tool is made for Mannequin's users first and foremost, so various features depend on it. For example, Mannequin cannot output models compatible with hand tracking at the moment, so we won't bother working on that feature in Puppetstring.
thanks for a quick reply, i didn't know at the time how to write an expression as opposed to a ratio in the tracking window or even what those words meant.
not sure where to put an error report, but puppetstring will capture the camera video but not display the wireframe that shows that it is identifying my face. I am on garuda linux (Arch based distro) if that helps.
Thanks for the info! I ended up changing distros in the end because i wanted something a bit more stable for streaming. I'll keep that in mind for next time though.v
comment: okay, I did some testing. note that not all bugs found were restricted to the use case or test case below.
use case: using keyboard controls to toggle emote blendshapes between 0 and 1 (key1..key9 toggling emote1..emote9) , accompanied by a separate group of keyboard controls mapped to the same key to set all emote blendshapes to 0 (a group of keybinds setting emote1..emote9 with key0). this would allow toggling independent emotes with key1..key9, and pressing key0 to turn off all emotes.
test case: keyboard control toggle; keybind (Alt)(1), blendshape "testShape", values {0,1}. keyboard control set; keybind (Alt)(0), blendshape "testShape", initial value {0}, set value {0}.
comment: Puppetstring 0.0.4 does not like it if you set multiple keyboard controls to the same blendshape. (that is, when several keyboard controls have their blendshape targets set to the same string.)
bug0: when a toggle keybind and a set keybind both target the same blendshape, the displayed values for the toggle keybind may appear to change correctly in Puppetstring, but the output blendshape communicated to session remains fixed regardless of keypresses. replicated successfully.
bug1: when a toggle keybind and a set keybind both target the same blendshape, clicking the edit button for the set keybind may instead result in opening the edit view for the toggle keybind, or opening the edit view for the set keybind as if it were an increment/decrement keybind. replication has been difficult; my conjecture is that the result of clicking the edit button for the set keyboard control may depend on variables or values that are persistent to a given instance of Puppetstring.
comment: as I was doing testing for this bug, additional bugs were discovered outside of the test case. keyboard set controls are hecked up, friendo. UI always goes like this lol
bug2: after editing a set keybind, future set keybinds may have an additional "Value 2" underneath "Key 1". for example, after three calls to edit any set keybinds, editing another set keybind might have a "Key 1" with the fields "Value 1", "Value 2", "Value 2", and "Value 2". partially replicated successfully; did not test whether or not the process is initiated by editing a set keybind which has multiple keys to set a blendshape to different values, or if it's initiated with any set keybind regardless of how many keys or values it has.
bug3: for a set keybind with multiple keys, the second key always sets the targeted blendshape to 0, regardless of the value entered for "Key 2" in its "Value 2" field. however, entering a value for "Key 1" "Value 2" will result in "Key 2" setting the blendshape to "Key 1"s "Value 2".
bug3 cont'd: "Key 1" "Value 1" {0.1}, "Key 1" "Value 2" {0.2}, "Key 2" "Value 2" {0.3}. this setup will cause "Key 1" to set the blendshape to 0.1 and "Key 2" to set the blendshape to 0.2. Replicated successfully.
bug4: editing a set keybind may result in all "Value 2" fields being initiated with the value from "Value 1", and the "Value 1" field being initiated as {0}.
comment: these are all the bugs I recalled and found in this sitting. there may be others I can't recall right now. and unrelatedto the bugs reported here, I have a suspicion there may be other bblendshapes like psHeadFrontBack where the values sent to Session may not match the values Puppetstring has, but I haven't gotten to around to actually testing this so I'm not confident whether this is the case nor which blendshapes may be affected.
final comment: best of luck, AR14. don't forget to hydrate, get up and move around, and find a bit of time to get some serotonin in.
(or maybe more like feature suggestions? idk, ignore these if you want, I'm not your mom)
(also sorry about the messy formatting, the comment field is deeply disagreeing with me)
keyboard controls
* toggles with transition times
* timed sets/toggles (automatically reverts to 0 after a user-defined amount of time passes)
* "held" keystroke type (sets a blendshape only while key is held)
* access to numpad keys
widget controls
* panel of tileable(?) widgets whose outputs can be mapped to custom blendshapes
* buttons (essentially keyboard controls but mapped to a UI object)
* 1D sliders/dials
* 2D touch pads (basically just a square you click in)
misc. blendshape management
* blendshape aliases
> if blendshape has arg1 name, send its value as arg2 name (maybe averaging values if multiple are aliased to the same name? or just let Session handle it)
* custom blendshape expressions
> custom blendshapes that are calculated from other blendshapes, using expressions (Javascript?) provided by the user
* tracking-activated toggles
> custom blendshapes that are set/toggled/incremented when a set of user-defined conditions are met (various blendshapes equal/greater/less than user-defined values or other blendshapes).
* tracking loss blendshape
> a blendshape that toggles to 1 if tracking is lost, or a "tracking lost" condition for custom blendshape expressions
I have use cases in mind for all of these, but also I'm not going to perish if 0 of them appeal to you lol
← Return to tool
Comments
Log in with itch.io to leave a comment.
Is there a way to download a version without any of the generative ai features? The idea of having them even if i won't use them feels icky to me, but this is the most recommended face tracker for inochi models.
I don't think so but you can try Inochi Session though you'll need to download face tracking softwares/apps. Face tracking softwares/apps usually has some form of AI but luckily some dosen't- not that I could find any with a quick google search but some outdated ones hopefully exsist, else, code your own. Most run using Google Mediapipe which has AI(unfortunately, it used to not have it but now it does) or others which usually has AI too. Face-tracking without AI at all runs on older, more traditional algorithms and hardwares which also require in turn alot more expenses even if you do find one without AI so I'd rather it include some form of AI. Traditional face tracking without AI require : Logitech C920s HD Pro Webcam(225SGD cheapest), 2x Micro servo motors(for pan-tilt, about 14SGD cheapest), Pololu 1350 USB Servo Controller(around 50SGD cheapest), Pan-Tilt Bracket Kit(36SGD cheapest), Ring lights, 3x soft lights, background light or greenscreen and of course the technical knowledge to put all that together including the pan-tilt system since most are hardware parts. Also the lights have to be neutral white lights, no color or gradients. Face tracking without AI requires specific specific hardware like those because it relies on tracking your face movements using geometric algorithms, etc which relies on high-contrast lighting in a dark room so the computer can calculate your face positioning and most hardware have AI chips nowadays. I'm by no means an expert so the information I listed may be lacking or incorrect and most of which are just from a quick google search but I do have a certificate for coding AI applications which was mandatory ALP(applied learning program) in my secondary school which was 5 or 6 years ago, closer to 7 years now(yes, AI has been around since way before the whole AI, Anti-AI thing.) As an animator, I do not support AI whatsoever but it's unavoidable. Vtube studio uses AI for face tracking as well so the only AI-free solution is to use Vbridger with a Live2D model which is extremely costly with Live2d models costing upwards 4k. Most hardware nowadays have some form of an AI chip though linux or linux mint is an AI free solution though certain games don't run on linux.
My problem is more the generative forms of at (Think LLM/Chatbots and image generation) and while i know i don't HAVE to use the provided chatbot, it just doesn't sit very comfortably with me just having it on my devices without my say. I appreciate the alternatives tho, it does help a ton and i probably will end up using an alternative.
Kinda curious if there if this is still being worked on and adding new features, cause its a handy tool, but could do with some added features.
It's still being worked on, however you should temper your expectation especially if you're not using it with models made by Mannequin Character Generator.
This tool is made for Mannequin's users first and foremost, so various features depend on it. For example, Mannequin cannot output models compatible with hand tracking at the moment, so we won't bother working on that feature in Puppetstring.
How do i simulate breathing with this? The Inochi sample model Midori and the bundled Clementine model do not breathe.
You have to rig the breathing first, you don't need this to simulate breathing as it can be done using Session only.
thanks for a quick reply, i didn't know at the time how to write an expression as opposed to a ratio in the tracking window or even what those words meant.
For those finding this thread later, how to breathe is on this page of the documentation (was hard to find as the homepage just says WIP) https://github.com/Inochi2D/inochi-session/wiki/Expressions
not sure where to put an error report, but puppetstring will capture the camera video but not display the wireframe that shows that it is identifying my face. I am on garuda linux (Arch based distro) if that helps.
Delete the `puppetstring` folder in .config if it's present. I've had this problem before and I *think* that's how I got it working again.
Thanks for the info! I ended up changing distros in the end because i wanted something a bit more stable for streaming. I'll keep that in mind for next time though.v
good news: I was able to get multi-instance puppetstring v21 working on both of my webcams.
bad news: my CPU did not appreciate the extra workload haha ORZ
comment: okay, I did some testing. note that not all bugs found were restricted to the use case or test case below.
use case: using keyboard controls to toggle emote blendshapes between 0 and 1 (key1..key9 toggling emote1..emote9) , accompanied by a separate group of keyboard controls mapped to the same key to set all emote blendshapes to 0 (a group of keybinds setting emote1..emote9 with key0). this would allow toggling independent emotes with key1..key9, and pressing key0 to turn off all emotes.
test case: keyboard control toggle; keybind (Alt)(1), blendshape "testShape", values {0,1}. keyboard control set; keybind (Alt)(0), blendshape "testShape", initial value {0}, set value {0}.
comment: Puppetstring 0.0.4 does not like it if you set multiple keyboard controls to the same blendshape. (that is, when several keyboard controls have their blendshape targets set to the same string.)
bug0: when a toggle keybind and a set keybind both target the same blendshape, the displayed values for the toggle keybind may appear to change correctly in Puppetstring, but the output blendshape communicated to session remains fixed regardless of keypresses. replicated successfully.
bug1: when a toggle keybind and a set keybind both target the same blendshape, clicking the edit button for the set keybind may instead result in opening the edit view for the toggle keybind, or opening the edit view for the set keybind as if it were an increment/decrement keybind. replication has been difficult; my conjecture is that the result of clicking the edit button for the set keyboard control may depend on variables or values that are persistent to a given instance of Puppetstring.
comment: as I was doing testing for this bug, additional bugs were discovered outside of the test case. keyboard set controls are hecked up, friendo. UI always goes like this lol
bug2: after editing a set keybind, future set keybinds may have an additional "Value 2" underneath "Key 1". for example, after three calls to edit any set keybinds, editing another set keybind might have a "Key 1" with the fields "Value 1", "Value 2", "Value 2", and "Value 2". partially replicated successfully; did not test whether or not the process is initiated by editing a set keybind which has multiple keys to set a blendshape to different values, or if it's initiated with any set keybind regardless of how many keys or values it has.
bug3: for a set keybind with multiple keys, the second key always sets the targeted blendshape to 0, regardless of the value entered for "Key 2" in its "Value 2" field. however, entering a value for "Key 1" "Value 2" will result in "Key 2" setting the blendshape to "Key 1"s "Value 2".
bug3 cont'd: "Key 1" "Value 1" {0.1}, "Key 1" "Value 2" {0.2}, "Key 2" "Value 2" {0.3}. this setup will cause "Key 1" to set the blendshape to 0.1 and "Key 2" to set the blendshape to 0.2. Replicated successfully.
bug4: editing a set keybind may result in all "Value 2" fields being initiated with the value from "Value 1", and the "Value 1" field being initiated as {0}.
comment: these are all the bugs I recalled and found in this sitting. there may be others I can't recall right now. and unrelatedto the bugs reported here, I have a suspicion there may be other bblendshapes like psHeadFrontBack where the values sent to Session may not match the values Puppetstring has, but I haven't gotten to around to actually testing this so I'm not confident whether this is the case nor which blendshapes may be affected.
final comment: best of luck, AR14. don't forget to hydrate, get up and move around, and find a bit of time to get some serotonin in.
okay, feature requests.
(or maybe more like feature suggestions? idk, ignore these if you want, I'm not your mom)
(also sorry about the messy formatting, the comment field is deeply disagreeing with me)
keyboard controls
* toggles with transition times
* timed sets/toggles (automatically reverts to 0 after a user-defined amount of time passes)
* "held" keystroke type (sets a blendshape only while key is held)
* access to numpad keys
widget controls
* panel of tileable(?) widgets whose outputs can be mapped to custom blendshapes
* buttons (essentially keyboard controls but mapped to a UI object)
* 1D sliders/dials
* 2D touch pads (basically just a square you click in)
misc. blendshape management
* blendshape aliases
> if blendshape has arg1 name, send its value as arg2 name (maybe averaging values if multiple are aliased to the same name? or just let Session handle it)
* custom blendshape expressions
> custom blendshapes that are calculated from other blendshapes, using expressions (Javascript?) provided by the user
* tracking-activated toggles
> custom blendshapes that are set/toggled/incremented when a set of user-defined conditions are met (various blendshapes equal/greater/less than user-defined values or other blendshapes).
* tracking loss blendshape
> a blendshape that toggles to 1 if tracking is lost, or a "tracking lost" condition for custom blendshape expressions
I have use cases in mind for all of these, but also I'm not going to perish if 0 of them appeal to you lol
where would be an appropriate place for feature requests/bug reports for Puppetstring?
You can just write it here or in Inochi2D's Discord server if you're already in it.