The talks with representatives from the Barisan Revolusi Nasional (BRN) insurgent group, expected to last one day, will focus on reducing bloodshed, Thai National Security Council chief Paradorn Pattanatabut said, warning the overall peace process would take time.
"Today's main focus is to reduce violence. Today we will focus on building mutual trust and good relations," Paradorn told reporters in the Malaysian capital of Kuala Lumpur, where the meeting was being held.
"I am confident that they will communicate our message to their militants but because BRN is a large organisation we have to give them some time."
The insurgency in the country's southernmost Muslim-majority provinces, which border Malaysia, has seen more than 5,500 people killed over the past nine years.
The meeting marks Thailand's first formal peace dialogue with rebels linked to the insurgency, which is being carried out by a number of shadowy groups.
Thailand blamed an early morning bombing that killed three people on militants seeking to sabotage peace efforts, underscoring the difficulties that lie ahead.
Three Thai paramilitaries were killed and five wounded in a roadside bombing targeting a security patrol in the southern province of Narathiwat.
"Violence this morning is related to the talks in Malaysia," Thai deputy prime minister Chalerm Yubamrung said in Bangkok.
He said not all militants supported the talks, reiterating the view of many experts who say there is a generational gap between older insurgents who want to negotiate and more militant younger members.
Chalerm also cast doubt on the BRN representative's authority to negotiate.
"I am not confident either they are real core leaders," he said.
Paradorn had said Wednesday that during the talks Thailand would seek to secure the halting of attacks on civilian targets so it can determine whether the BRN envoys actually control battle-hardened militants on the ground.
Little is known about the various militant groups' identities, structures or aims.
There is near-daily violence in southern Thailand including bombings, ambushes and even beheadings.
The BRN, whose Malay name means "National Revolutionary Front", is one of the larger groups held responsible by Thailand for the violence.
Paradorn said Thailand hoped other groups would join future talks.
"I see BRN as the largest and most influential group behind the unrest, so after the talks BRN can spread the word to their fighters and then we will see a concrete result," he said.
Muslim-majority Malaysia has already hosted negotiations between the Philippines and Muslim separatists in that country which resulted in a landmark agreement in October aimed at burying a decades-long insurgency there.
The roots of the Thai insurgency draw on long-standing Malay nationalist antipathy to rule by Buddhist Thailand, which started when Bangkok annexed the region in 1902.