Your pseudo code uses non standard notation (you seem to be using assignment in the <= comparison expressions, that is really confusing), which is makes it hard to read, but If I understand the issue correctly, its that there is some confusion with friction.
Friction in real life is caused by microscopic perturbations in the surface of a plane an object is standing on, characterized by a friction coefficient for simplicity sake eg: u. The magnitude of the force of friction is this (see wikipedia for an illustration):
friction_magnitude = u * magnitude(force_anti_normal_to_friction_plane);
A plane can be arbitrarily oriented, not just flat horizontal. But see that we use force_anti_normal_to_friction_plane and in your equations, you appear to only care about the acceleration due to gravity, when the force equation is actually F = M*A. The mass is accounted for in the force of friction. In this way your friction magnitude assuming gravity is the only force normal to the plane of friction should be:
friction_magnitude = u * magnitude(G * mass);
You then take this and multiply it with the part of the velocity parallel to the plane of friction, for a horizontal plane this is just your non vertical components (in my system this is normally x, and z) normalized.
friction_force = -u * magnitude(G * mass) * normalize(vec3(velocity.x, 0.0, velocity.z));
In the case where you have an arbitrarily oriented friction plane it is the rejection of the velocity vector with the normal of the plane which contains the components which to normalize, and the projection of the force to figure out what components of the force actually apply to the plane of friction. This rejection contains the orthogonal vector to the normal of the plane, which contains the parallel velocity component to the plane, the projection contains the components parallel to the the given vector.
// need to guarantee that the forces are at least pointing towards the plane
// and not away, if forces do not point towards plane, then there is no
// force anti normal to the plane of friction pressing down to cause contact
if(dot(forces,friction_normal) < 0.0){
force_anti_normal_to_friction_plane = projection(force, friction_normal);
}else{
force_anti_normal_to_friction_plane = 0.0;
}
friction_force = -u * magnitude(force_anti_normal_to_friction_plane) * normalize(reject(velocity, friction_normal));
Another thing you need to watch out with dynamic friction is that you don’t want to apply it past when the object would have stopped moving in the parallel velocity to the plane. Ie if you have a very tiny vector moving across the floor of your friction plane, friction doesn’t scale less because of that, and doing a naive calculation would cause your velocity vector to move back and forth forever across the plane. You need to halt your movement along that vector to adequately handle this case. In otherwords, if the magnitude of your velocity along the plane of friction is less than the magnitude of velocity change contributed soley by the force of friction you need to stop moving!. To do this for arbitrary orientations of the friction plane, take the velocity components rejected by the normal and subtract them from the velocity to cancel out that movement.
