Does the bard’s Jack of All Trades feature apply to attack rolls and saving throws that don’t use the bard’s proficiency bonus?

No. The feature benefits only ability checks. Don’t forget that initiative rolls are Dexterity checks, so Jack of All Trades can benefit a bard’s initiative, assuming the bard isn’t already adding his or her proficiency bonus to it.