* More than 5,000 have died in violence
* No details on whether ceasefire likely; talks start in 2 weeks
* "Major milestone" in move to peace - analyst (Adds details from Thai, Malaysia PM statements)
By Siva Sithraputhran and Stuart Grudgings
KUALA LUMPUR, Feb 28 Reuters) - Thailand's government will hold formal peace talks with a major Muslim rebel group for the first time next month, marking a breakthrough in efforts to end a worsening conflict in the country's south that has claimed over 5,000 lives since 2004.
Senior Thai officials and members of the Barisan Revolusi Nasional (BRN) group signed the agreement for the talks in the Malaysian capital of Kuala Lumpur. However, there was no immediate word whether it would be accompanied by a ceasefire, following an upsurge in violence in southern Thailand, near the border with Malaysia, in recent months.
Initial talks between the two sides with a senior Malaysian official as mediator would begin in two weeks, Malaysian Prime Minister Najib Razak said in a statement following a meeting with his Thai counterpart Yingluck Shinawatra in Kuala Lumpur.
It was just the start of a "long process," Najib said, adding that peace would bring prosperity to both sides of the border.
"Let us hope that all parties will respect the process," Najib said.
A Malaysian official said the initial meeting would aim to establish terms of reference for the talks and that the BRN had been chosen because it was the dominant rebel group.
Successive Thai governments and the military have made contact with various rebel groups and are believed to have held secret talks, but they have never had open discussions.
The talks follow a shift in Thailand's stance last year when it acknowledged the conflict's "political nature" for the first time after years of relying on military action that has steadily alienated majority Muslims in the southern provinces.
"This is a major milestone," said Anthony Davis, a Thai-based analyst at security consulting firm IHS-Jane's. "This is not just business as usual. This confers a level of legitimacy on the armed opposition in southern Thailand, from which realistically there is no going back."
Past efforts to end the conflict have led nowhere and the legitimacy of the new process could be thrown into doubt if some rebel factions respond with more attacks in coming weeks.
A peace process could also face institutional resistance from Thailand's military, which has 60,000 troops in the south enforcing a state of emergency.
"The signing is part of a policy written by the government. The military has nothing to do with today's decision," Udomchai Thammasarorat, commander of the Fourth Army in southern Thailand, told Reuters.
"We've opened up other ways to begin dialogue including giving insurgents an opportunity to turn themselves in and fight their case through the Thai justice system."
Resistance to Buddhist rule from Bangkok has existed for decades in the predominantly Muslim provinces in the south, waning briefly in the 1990s before resurfacing violently in January 2004.
In recent months, the shadowy rebel groups have stepped up their campaign against schools in the area, which they see as a state tool to assimilate Muslims, killing at least five teachers.
Sixteen insurgents were killed in an attack on a Thai marine base on Feb. 13, with no loss of life among the marines. Rebels hit back with a string of counter-attacks including an explosion in Pattani province that killed three security volunteers.
MALAYSIA'S PEACEMAKER ROLE
The agreement promises to burnish Malaysia's role as a regional peacemaker after it helped broker a historic deal between the Philippine government and Muslim rebels in October.
Security analysts say the ethnic Malay insurgency has long relied on safe havens and supply networks across the border in northern Malaysia. Those suspected links, which Malaysia denies, have periodically strained ties between the neighbours.
Peace in the Thai south would remove a glaring obstacle to regional plans to improve rail and road links in the fast-growing 10 economies of the Association of Southeast Asian Nations.
Insurgents have frequently targeted the railway system, killing two people with a bomb under a train last November. Tourism also suffers. Malaysians visit from across the border but other nationalities generally steer well clear.
Thailand is the world's biggest rubber producer and the bulk of it is grown in the south. Insurgents have occasionally targeted plantation workers, beheading some, although the impact on the rubber trade has been muted.
Thai authorities say the attacks in the south are organised by the BRN Coordinate, an offshoot of the Patani Malay National Revolutionary Front established in the 1960s to seek greater autonomy.
"We'll do our best to solve the problem. We'll tell our people to work together to solve the problem," said Hassan Taib, who signed the document in Kuala Lumpur and was described as the BRN liaison officer for Malaysia.
Hassan was identified as "a senior separatist figure" in a December 2012 report by the International Crisis Group, a Brussels-based think tank. (Additional reporting by Andrew Marshall and Amy Sawitta Lefevre in Bangkok; Editing by Raju Gopalakrishnan and Alan Raybould)